Martin Creaner’s blog teaches us the fundamentals of cloud computing, but where's the TM Forum leadership?
The hype and hope of cloud computing continues. There's an opportunity here, and I'm happy to see more contributions to the debate on how service providers can embrace new architectures. Frequently, though, they lack any meat, or real understanding of what cloud computing actually is.
above. Cloud computing architecture. Click to enlarge
Martin Creaner, President of Tele Management Forum, has recently blogged his two pennies worth. He's somebody senior management listen too, so it's good to see he's written a piece that brings clarity to the whole cloud thing.
Take a look at Martin's article on the TMF blog.
But What About The Cons?
Martin is a little too convinced that cloud computing will be the Next Big Thing. There are plenty of potential 'fatal flaws' in the concept that deter users today and could easily scupper the movement in the next few years. Just two of them:
No portability. You can't pick-up a Google AppEngine application and reinstall it in Amazon EC2 or Microsoft Azure. Each cloud platform is proprietary. Really proprietary. If you thought supporting applications on Solaris and Windows Server was a hassle, this is nothing compared with cloud computing.
Limited Functionality. OSS and BSS applications are non-trivial. They rely on complex business logic, workflow engines and sophisticated features of relational databases. Most of this is not available in the cloud. You get Java POJOs, a component model, and a flat data store that's got less functionality than Excel.
It may just be a matter of time before these problems are solved. We're starting to see Java support in clouds, and some platforms, like Azure, are expected to introduce proper SQL databases. They better get a move on though...
Here's What the Cloud Can Do For You
The high-light of Martin's piece is his clear explanation of the benefit of cloud computing: You get the resources you need quicker, and pay less up front for them, compared with procuring your own servers and software.
These simple financial savings and time savings are the basis of the cloud proposition. Maybe that's a little disappointing? Were you expecting it to be whizzy-bang technology that revolutionized the way you did X, Y or Z? No. Cloud computing's strength is that the fundamentals are simple, measurable, benefits.
above. Benefits of cloud computing. Click to enlarge
More, Please, Martin
So, Martin's a cloud computing fan. This needs to lead in to some technology leadership from TM Forum. Sorry, but despite Martin's suggestions NGOSS, or any other SOA initiative, is not an enabler of cloud computing. SOA is architecture agnostic, that's the point of it, so having SOA contracts does not make implementing a specific back-end architecture any easier.
The potential, revolutionary, use of clouds is, as Martin surmises, in building a scalable architecture for Telco 2.0-style upstream partners to 'plug-in' to the network and OSS/BSS infrastructure. Partners and service providers can then go mad and deliver the massive suck-it-and-see approach to products that Keith Willetts (TM Forum) has been talking about lately. With the economics and scalability of the cloud, entry costs for new products and partners is small. Scaling-up a surprise success is affordable and, most importantly, possible.
To make this a reality we need APIs, contracts, SLAs and processes that allow partners to run their businesses through service provider infrastructure, much like retail vendors do today on the Amazon platform.
TM Forum would be doing the industry a big favor, while restoring its reputation for innovation, if it delivered a specification for a model Telco 2.0 infrastructure.
I was catching up on some Marketing Over Coffee podcasts today when the topic of Web 2.0 ‘single points of failure’ came up briefly. Web 2.0 stuff is great for making contact with customers and building interest in campaigns. MOC questioned whether such services could be relied on, 24/7? Their marketing answer was to ensure you back stuff up and make the most of traditional customer relationship tools too.
It got me thinking about the implication for service providers battling over-the-top services.
“Over the top: Digital industry term describing third party home entertainment services that are delivered across (i.e., "on top" of) a broadband network without affiliation with the broadband service provider.” – Wikipedia.
Consider Twitter, the example MOC used in their podcast. A business may rely on Twitter for customer feedback, or as the basis for selling some cool value-add service (like TweetDeck). Consumers may use Twitter instead of spending money on phone calls and SMS. But these users run the risk of two types of single-point-of-failure: System failure and commercial failure.
System Failure
Twitter has fallen over in the recent past. It became renowned for its iffy architecture. Even the mighty Google, with its clouds and data centers, has lost some services from time to time. Web 2.0 services typically have a single service provider, maybe a single data centre, application instance, that will fail at some point, even if they use reasonable internal redundancy in their hardware and network platforms.
This is quite different from email, for example, which is very resilient to failure. Each organization, or at least each ISP, has its own SMTP, POP3 and IMAP services. These communicate over the Internet which relies on telco-grade networks. A well built, well managed network, running an inherently resilient protocol, with multiple service nodes.
Commercial Failure
What if Twitter goes out of business? Given the lack of obvious revenue streams for many Web 2.0 services, this is not at all unlikely. Without clear revenue streams it would also be quite possible that no one would step in to take over. The industry would just let it die as a failed experiment.
And you can’t just move your Twitter account to a competitor, if any are left. The technology is different. Integration APIs are different. Assets such as URLs may have to change. Groups, followings, friends and so on would have to be re-established.
If Twitter, then why not Skype? If Skype went down the pan, what are your options for number portability to another service provider?
Traditional service providers have clear revenue streams. They have physical assets. They have intellectual assets. Sure, a telco could crash and burn if the economic situation was critical, but it’s much less likely that no-one would take on the business. And thanks to deregulation, it would be easy to move customers from a failed service provider to a healthier one.
Cheap Cheap!
‘Go telcos!’. They’re clearly better, right?
Over the top services are attractive to end users. They’re innovative, colorful and cheap to use. Despite all the hype around ‘customer experience’ it is often the customer who is last to consider their own overall experience. Web 2.0 is instantly accessible and instantly satisfying. Customers, use to stuff ‘just working’, are happy to assume Web 2.0 services will satisfy their medium and long-term needs, at least until something else shiny comes along.
Web 2.0 will only be a success (i.e. have a practical revenue model) when businesses and customers have a long-term financial or lifestyle investment in the service. Only then will reliability and stability start to be a bigger factor, in the customers’ experience, than instant satisfaction.
Until that day of judgment comes, when traditional telcos are vindicated and hailed as champions of quiet efficiency and reliability, customers will put up with the experience of free, easy, no-strings-attached, blissful naivety.
Aaaah, blissful naivety. Don’t ruin that for us, just yet…
Back, oooh, ages ago, I wrote about the top five OSS platform architecture things. Those where things (issues, concerns, designs, e.t.c.) that needed to be considered when delivering a new OSS product or project to ensure scalability and performance.
Rarely does a new OSS application live in isolation. Integration with other systems in the OSS and BSS environment could easily be as much work as the development of a new application.
It’s raining today, making me feel a bit low, so I’m going to give you five anti-patterns for how not to integrate your OSS/BSS applications.
1. Opening APIs to Remote Calls
2. The EAI Silver Bullet
3. Avoiding Asynchronous Interfaces for Performance Reasons
4. Data Without Context
5. Implementing Standards
Opening APIs to Remote Calls
It’s now very easy, almost automatic, to take a reasonably well designed application and expose its APIs to remote calls for integration purposes. For example, a Java-based application may have a set of internal interfaces to create/read/update/delete objects in its database. These Java Beans were designed for use by the application's UI or business logic. Wouldn’t it be nice to offer them as an integration API? Easy. Just point your development tool at the Beans, press a button, and you’ve got a remote API using RMI, SOAP, CORBA or whatever.
Java EJB makes it so easy as to imply it should be done, that any piece of code could reasonably be used locally (internally) or remotely. There are lots of reasons that doing so would be a very Bad Thing.
Such APIs are too fine-grained to be useful. An interface specification that comprises dozens of separate Java classes isn’t going to be popular with your system integrators, particularly five years down the line when some other SI has to work out how to use the API calls to actually achieve something meaningful. A fine-grained interface does not encapsulate any business logic. It doesn’t add any value.
The API will require a ‘chatty’ interface. Many API calls would be needed to achieve something useful. Such integration is known as ‘chatty’, as many messages have to pass between the integrated systems to get a relatively simple job done. Remote interfaces, even using ‘low-level’ interfaces like CORBA and RMI, are *very* slow. An order of magnitude slower than the internal calls the application makes to its own local interface. Chatty, slow, remote interfaces are obviously to be avoided.
System integration will be too tightly-coupled. The external system, or adapter, will need to use very specific, low-level API calls executed in precisely the right order to achieve something meaningful. If the applications data model or internal processes ever change (and they will) then the low level API will most likely also change, effecting any external systems that used its remote interface. The cost of maintaining the integration over time will be higher than the initial development cost.
The EAI Silver Bullet
Seven years ago, or there-about, people were struggling with nasty, low-level CORBA-based integrations. They were even being promoted as industry standards. They weren't particularly successful.
Then, a little later, middleware vendors provided the answer, as they so often do! They gave us enterprise integration in the form of various TLAs (ESB, MOM, but I'll use the more general Enterprise Application Integration, EAI, name). The plan was to migrate away from all the multiple, low level, tightly coupled integrations and move all system integration on to a reliable, manageable, homogenous hub or bus architecture. It did look good on paper. Spaghetti-like system diagrams became clean and lasagne-like. But, if people ever thought 'we'll use EAI for everything' (and they did) they were making a big mistake.
Most integration is still point-to-point. Asynchronous, publish-subscribe integration has its benefits, but the reality of most OSS/BSS environments was that the applications are not optimized for this sort of integration. Legacy, and many new applications, still required the EAI to implement spaghetti-like point-to-point integration.
All those nice enterprise features come with a performance hit. I like all that guaranteed-message-delivery, management consoles, XML payloads, message transformation, and stuff. But it all uses up CPU time and/or network bandwidth. Often it's worth it. But for very quick, point-to-point, low-latency, high-volume integration EAI is not fit for purpose.
It can become a thorny issue when a project mandates use of EAI, but the delivery team is forced to bypass it. Occasionally, one has to accept that there are good (demonstrable, measurable) reasons to use something other than the incumbent EAI.
Oh, by the way, SOA isn't this decade's silver bullet either.
Avoiding Asynchronous Interfaces for Performance Reasons
Did I just say that EAI asynchronous messaging buses offer poor performance? I don't think I said that. Just sometimes you need an integration technology that's an order of magnitude faster.
People have been stung by trying to use too much EAI, and go to the other extreme of assuming all asynchronous interfaces should be avoided when performance is critical. This removes a useful option from the system integrators toolbox.
Here’s an edge-case where this could cripple overall system performance: When evolving a complex OSS/BSS environment some legacy systems expect very quick acknowledgement to their external requests. If one replaces the external system with a newfangled application that maybe doesn’t respond so quickly (but meets its own performance requirements) and uses a slightly slower interface technology (say, replacing system-level pipes with SOAP) you significantly increase the request-acknowledge latency. This may be acceptable, except for the impact it has on the legacy system’s concurrency: If the system locks any internal resources while waiting for an acknowledgement of its request, any increase in latency can be a serious problem.
Using asynchronous messaging easily solves this problem. The request message reaches the external system, get an immediate acknowledge and is placed on a queue to be processed at the external systems’ leisure. Asynchronous messaging can de-couple decoding and processing from message delivery. When you just need to know it got there, asynchronicity gives you a fast response time.
Data Without Context
Knowledge and wisdom, data and information. There’s plenty of clichés on this topic, yet its easily overlooked when it comes to designing OSS/BSS system integration. The assumption seems to be if you get the data integration or synchronization right, then that’s 80% of the job done. No. Data is 20% of the job, and the context and meaning of the data is the 80%.
There is no single ‘correct’ view of the network that is universally useful. Yes, data reconciliation to an inventory, for example, is useful for activities such as service provisioning, fault management, and so on. But, my point is, an overall OSS/BSS system design has to recognize that different applications need different data and information.
There’s little value in putting the integration effort in to creating one view of the OSS/BSS world. Take a network device, for example. An activation application may see a device with logical and physical interfaces. A provisioning inventory, might have this plus a simple shelf-slot-card data model. An inside-plant tool would have a complex physical data model, but no logical data. While in theory the common elements modelled by these systems should be identical, and therefore candidates for inclusion in a centralized repository, they are rarely easy to recognize. Differences in naming conventions are common, causing significant integration headaches, and also usability issues: which is the ‘right’ name for the centralized repository? Then there are technical issues like finding unique keys to align the data and managing mapping between subtly different device models (maybe tricky issues like card-in-card or aggregate ports are modelled very differently in each system).
All things considered, when is a single centralized data repository worth the effort? Umm… Can’t think of one. That’s why the idea put forward five years ago by EAI vendors for data-on-the-hub came to nothing.
Data needs context to be useful, so inevitably several OSS/BSS applications will maintain their own view of the world. This view will need some synchronization, which requires the context to be understood and maintained by the synchronization process. How does context vary between applications?
Data time frame is a good example of context. Each application considers 'current' to be something subtly different. For an activation of alarm application, 'current' is the last message from the network. For an inventory system, such transitory states may be irrelevant, and a network device 'current' is a record of its design, as implemented in the network. For planning, 'current' may encompass the state of the network after the current month-long roll-out, as this is the design that 'future' builds are based on.
Data integration is therefore not a case of comparing two XML files for differences and updating a database accordingly. It is only of any value if, first of all, a well understood process integration is taking place.
Implementing Standards
Standards organizations are A Good Thing, but their output isn’t always as useful as it first appears. People have a tendency to misinterpret the intentions of standards bodies and that’s when bad design decisions are made.
Back in the day, when I was a sales engineer, I was cross examined by a prospective customer regarding the standards we had used to implement our inventory. I had discussed at some length the design of the database schema (a bit low level, yes, but they get an idea for its sophistication and integration potential) but a serious question was raised: “Why doesn’t your database implement the SID standard?”. Assuming the true question was lost in translation I responded by explaining they certainly could have SID-type objects in the inventory, and import/export adapters could be offered. “No, why aren’t your database tables based on SID?” And at that point I realized any discussion about good application design, flexible modelling, and data modelling IDEs, was probably going to get me nowhere.
SID, has its uses, I’ve seen successful trials of 3GPP network inventory data standards, and I quite like MTOSI. But none of them tell you how to build an application. They are contracts, file formats, concepts and common reference points. Applications are not standards-compliant, but their interfaces and business processes can be. When implementing standards do so in a light-weight way. Use the standard as a target requirement (‘we need to store and exchange MTOSI data’) but not a strict blueprint for implementing data models and code.
Anti-Pattern Number 6: Listening to Other People
The biggest mistake in any IT project is taking peoples opinions at face value, particularly if that person claims to be some sort of expert (or blogger).
With OSS/BSS integration you’ll find yourself in unique situations where integration of the latest technology has to co-exist with legacy system integration that's been in place for 10+ years. This means there is no single answer, no single pattern for integration. You might be able to implement SOAP, REST or EAI in some areas, in other you’ll have to make a strategic decision to use the existing string-and-chewing-gum integration, no matter how architecturally impure that may be. If it works, it works.
All those cool technologies out there, I wonder how any young developer will have space for them on their two-page curriculum vitae. And it's not just the quantity, it's the names. Languages, operating systems and middleware used to have nice short names and acronyms. Concise, optimised, no more than four characters long. But now we have “Ruby on Rails”.
above. Choose the right tech for the job.
There's a trend here: The longer the name of the technology the more concise the language and the shorter the development time. At least, that seems to be what's proposed by many new technologies.
The emerging languages don’t cut the mustard as far as product development goes. I wouldn’t want to support and evolve a product for the next 20 years and use anything but a well established enterprise-grade platform. But that said, sometimes a cool new technology can satisfy immediate needs for a tactical or short-term solution.
The Register ran a report on Twitter's migration to a new system architecture. Reportedly because Ruby on Rails just couldn't hack it. It scaled up to a point then just fell over, famously during Jobs' keynote at a Mac show. To be fair, a massive burst of traffic can bring almost any system down with the right set of conditions. However, Twitter recognised that there was a general scalability issue here, and Ruby was not going to allow their platform to scale along with their business ambitions. So, on the server side, they've swapped Ruby for a little known, concise, language with no proven enterprise track record, called Scala. Okaaaayyy... Let's hope they benchmarked it this time...
Ruby, Scala, Java, ASP.NET, C++ and AMOS Basic all have their place and valid uses. Pick one that’s appropriate for the job, and make sure you prove it can meet your requirements.
above. AMOS Basic – The right architectural choice for bedroom coders in the 1990s.
If you're working on a new OSS system you should quickly narrow the choice based on the fact that there are three categories of technology out there:
Suitable for enterprise-class OSS products. A commercial off-the-shelf (COTS) product's primary non-functional requirements will be portability, scalability, configurability and manageability. Manageability, if it is a real word, means basing the product on technologies that have good support services, release cycles, flexible licensing terms and limited inter-dependencies. You need to be sure your platform components are thoroughly tested on a variety of enterprise servers, and will be supported and maintained for years to come. Broad industry adoption provides some reassurance that the technology will continue to be invested in for the foreseeable future.
Suitable for enterprise-class OSS projects. If you're developing a point-solution, something that has horrendously short delivery timescales and will probably be hacked around every other month, you might as well use any relatively proven technology, whether that be Ruby on Rails or some drag-and-drop Java tool. Rapid development is the priority, provided a stable technology is used, but as its built to a fixed specification and a single known environment the concerns around delivering a managed COTS products are far less relevant. Not every project is led by time-scale. If you're producing something to be used throughout a large service provider that must support network/service evolution then you're effectively writing in-house products.
Other stuff. Some technology is not sufficiently proven to be recommended for anything but the odd proof-of-concept or small tactical intranet application. Other technologies are starting to fall into the gap between enterprise platforms and emerging technology. PHP, for example: Would you start a new project using PHP rather than going with either Java or cool/hip/funky Ruby on Rails? No, PHP is soooo four-years-ago.
Twitter was developed like a small project to deliver a cool little application. I don't think for one minute any product managers sat down to discuss functionality and non-functional requirements. It's just not the romanticised web 2.0 way, is it?
I like new stuff, I really do. But when it comes to using something for an OSS product it better be scalable and robust first. Concise, agile, and rapid would all be on my wish-list, but not near the top. Maybe that's why my CV fits on two pages.
There are a number of ‘standards’ out there that, as a system integrator or product vendor, you may want to use within your OSS solution. Stuff like MTOSI, OSS/J, SID, 3GPP all look like useful starting points for inventory-like data models and their integration points.
It is important to appreciate the difference between a standard’s published specification and an implementation of a useable database model and business logic.
Let’s consider TeleManagement’s MTOSI as an example, as it is the most popular standard for network data exchange right now. The MTOSI specification is not concerned with implementation of an inventory model, or how it is maintained by a database and operated against by an application. Nor does MTOSI define ‘metadata’; the definitions of what are valid models, e.g. what equipment comprises a specific type of Cisco router. MTOSI just provides a means of describing network elements. MTOSI arose out of a need for a standard way of pushing up the data held by network element managers in to network management systems or cross-technology OSS platforms. So, it’s main focus is in defining a data format and a number of operations around retrieving or modifying the data.
Here’s a very high level picture of the MTOSI physical model. There are many other entities that can be modelled, and much more detail in the actual specification, available from the TeleManagement website.
above. Simplified MTOSI Physical Model. Click to Enlarge.
Looks good. If I’m developing any application that needs its own internal record of network equipment, this would be a great place to start.
MTOSI’s physical equipment model is quite straight forward. It allows hierarchies of objects to define the common device-shelf-slot-card-interface model of telco equipment.
To some extent the MTOSI model looks like a database model, as if it could be mapped directly in to tables to create a useful inventory. This is not the case, for a technical and design reasons.
Object-Relational Mapping. The MTOSI model is an object oriented model. Databases are usually relational model. It’s the old ‘object-relational impedance mismatch’ that requires applications to do apply business logic to an object model but persist those objects in a relational database, such as Oracle or MySQL.
There are also OSS and application-specific design considerations that will mean the MTOSI model may be best implemented in a particular way. These three factors normally have to be balanced depending on the OSS application’s requirements. For example one might have to sacrifice some performance in favour of flexibility.
Design for Performance. Response times, concurrency and scalability all combine in some acceptable compromise to deliver an application that 'performs well'. In OSS, raw single-thread performance is usually traded-in for good concurrency (multiple process/user threads) and scalability (effective use of additional hardware). Overall performance is usually best when building data schema and code to a static design, i.e one that does not rely on introspection and data-driven code. A static design allows for assumptions to be made, reducing checks, queries, lookups, error handling and process roll-back overhead.
Design for Flexibility. Some applications work to an original, fixed design. Rarely is this true of OSS applications. In OSS flexibility must be offered to allow customers to deploy their own inventory objects, provisioning processes and activation scripts. If done well, flexibility is enabled through data-driven configuration and clearly defined APIs. Architecting these features requires extra layers of abstraction and checks which inevitably result in reduced performance.
Design for Maintainability. While the end customer needs application flexibility to make changes to their deployed product version, the software developer needs to be able to maintain the application as requirements change between product versions. For example, an inventory product based on the MTOSI 2.0 standard will one day need to be upgraded to support MTOSI 3.0. This could either require a redesign of the data scheme, possibly impacting backwards compatibility, or it could be achieved through metadata and limited schema changes supported by clever 'introspection' by the system's public APIs.
At one extreme, systems that have critical performance requirements, such as data migration or bulk update processes, would be designed for high performance at the expense of flexibility and maintenance: assumptions might be hard-coded in the process, integrity constraints disabled, layers of system architecture replaced by transaction scripts. A multi-purpose OSS inventory, that needs to support several applications and business functions, would expect to sacrifice performance in favour of flexibility: Layers of system abstraction, configuration, metadata and integrity constraints would be implemented.
above. Design Requirements for OSS Applications. Click to Enlarge.
When implementing a standard, designing database schema and coding business processes, one has to start by establishing what the priorities are that will drive the system architecture design.
> The standard’s specification itself does not dictate how you implement it, only what data and services it should support.
> There’s many, many possible ways of coding a database and Java application for any one given standard, such as MTOSI.
> An implementation of a standard will vary depending on the type of application that is using it.
>The first question to answer is, how important is performance, flexibility and maintainability? They can’t all be high priority!
Coming soon! Part two will look at options for mapping an MTOSI-like standard to a relational database such as Oracle, considering common architecture options and their suitability for particular OSS applications.
OSS and BSS are data-centric systems, as I am sure you appreciate. Billing records, CRM records and inventory data immediately spring to mind as the classic data silos found in any service provider. Despite the growing sophistication of both OSS and BSS solutions, the problem of managing and accessing this data has got worse in the last ten years, not better.
There are more types of services being sold, often based around a separate silo of OSS systems. This is particularly true since the spate of acquisitions that has meant either parts of the service providers’ geographies or sections of their network technologies each have separate data silos. If you take a look at the OSS/BSS of a large national quad-play service provider you will probably see:
> Silos of OSS and BSS stacks under one or more of the quad-play services.
> Swivel-chair CRM and physically separate technical support.
> Product and service catalogs distributed around OSS and BSS silos.
> Multiple legacy inventories and next-gen commercial inventories.
> Multiple activation platforms distributed by region and technology.
For the last couple of years OSS vendors have promoted ‘transformation’: Migration away from legacy silos to a unified environment that supports all their next-gen quad-play service delivery needs. This would be lovely. Some providers are even doing it, but they are in the minority. A tier-2 or tier-3 provider might achieve transformation due to the relatively small scale of their requirements. A tier-0 may undertake the project because they’ve got deep pockets and massive operational pain from their acquisitions and mergers. Not every telco has the stomach for transformation and its significant costs, and those that are on the road to unified OSS have to accept it’s a long journey.
What then for the untransformed? And for those unified-OSS butterflies, still in the chrysalis for another three years?How do they get at their fragmented data?
Other than transformation, two other approaches to fixing the data silo issue have emerged recently: Federation and Enterprise Search.
Federated Inventory
Federation is the hard-core solution. It deploys a data-access, and sometimes process-access, layer across multiple silos. It will then either aggregate data through synchronization to its own repository or store only local references to external objects, then query the target data sources on-demand for detail. A commercial OSS inventory system can then be deployed on top of the federation layer. To the end user, and the inventory application for the most part, the appearance is that of a single OSS data source. Under the hood the federation layer is managing read/write activity across the multiple data silos. To allow an OSS application to operate on external data sources as if they were local significant integration plumbing and data transformation is needed.
Are there any commercial OSS federation products out there today? Sort-of-ish. There are a few products that are more like classic data synchronization/reconciliation (e.g. Telcordia’s Data Federation & Analysis offering, and PSI’s MCCM tool for mobile data reconciliation). And there are federation products that work with a narrow set of data rather than all OSS/BSS data sets (e.g. Amdocs Unified Service Manager). Grand-scale federation is the reserve of ambitious service provider projects, for the time being at least.
Enterprise Search
Enterprise search is the light-touch solution. Effectively a data access layer is deployed across all data sources, accessed through its own UI and API. A single UI provides ad-hoc browsing of data and the ability to build reusable views appropriate to particular business activities. Source data is ‘crawled’, indexed and cached to make the search process much faster than it would be if the data was directly accessed for each user operation. Enterprise search does not attempt to replicate other application's functionality. As a result its data model is more relaxed and integration effort is reduced by reuse of gateways that know how to crawl and relate data in common repositories such as databases, spreadsheets and XML files.
The key distinction is obvious: Federation attempts to deliver OSS functionality across multiple data sources, while Enterprise Search aims to provide a unified view of the data through an additional application.
NexGenData: Enterprise Search and Web 2.0 for Telcos?
Scott Kelly and George Schaefer from NexGenData gave me the opportunity to check out their Navigator enterprise data integration product. NexGenData is a start-up with a strong telco and OSS heritage having contributors with a background in Visionael and EDS, among other organizations. So while their enterprise Web 2.0 technology is essentially data-agnostic, it is being designed with OSS/BSS applications in mind.
The primary value proposition is a significant reduction in the ‘integration tax’ compared with middleware, federation or data warehouse solutions. Once pointed at a few data sources NexGen Navigator is able to crawl structured (e.g. database, XML) and unstructured (e.g. text, HTML) content and build up relationships. For example, a customer ID or simply an address could be used to tie together records from multiple CRM, billing and support sources. And that’s the key benefit of search technology: Data integration is part of the out-of-the-box algorithm, rather than something that has to be explicitly coded on a case by case basis.
With the crawl of multiple data sources completed, the user has access to what looks a lot like a typical OSS inventory tree-level browser. Data can be searched and then ‘drilled-down’ to uncover hierarchies and relationships.
A couple of other Web 2.0 principals are employed to further enrich the data and user experience:
> Mashups allow users to merge a collection of data feeds in interesting ways, sharing the results as standard web report URLs or as input in to more mashups.
> The grandly named ‘collective intelligence’ feature is like delicious for enterprise data. Users can tag useful and important data streams to promote their ‘usefulness’ to other users. For example, data streams that relate to an important customer could be tagged, promoting them when presented to other users searching for data on that customer.
NexGenData are targeting large enterprise customers, government and service providers with Navigator. Generally speaking, the larger the organization the greater the problems. But it also depends what business processes touch the various data silos. Telecoms is more prone to data silo issues than most as many of its business processes need to cut across department, commercial, and technology divisions. OSS has always tried to address the need for a coherent view of the entire network data source. In the BSS area CRM attempts to provide a unified picture of the customer and services. But if OSS and BSS data source are not joined up, and if silos of data exist in these areas, the costs to a service provider can be significant. George and Scott quoted one US tier-0 that believes misalignment of siloed data causes $30m-$50m in costs per month. Fifty million Dollars per month. Crikey.
To support service providers, Navigator must offer good scalability to deal with the dozens, maybe hundreds, of data sources these organizations will have, and several million data objects contained there-in. Fortunately, we know search technology does scale; it does a pretty good job of indexing the web with its many data sources, distributed systems, poor network latency and high user count. NexGenData have applied some proprietary IPR to simplifying the issue of scalability for their customers. They provide Navigator as an appliance, a single box solution in the first instance. Additional boxes can then be added to create a cluster of boxes supporting a single Navigator deployment. This should satisfy two architectural requirements:
> Scalability. The ability to add more hardware to increase overall system performance.
> Distribution. The need for the appliances to located in different data centers to satisfy IT network requirements, remote access latency issues, or physically diverse resilience.
Data Context
Enterprise search is starting to spawn a number of start-ups offering the ability to access generic corporate data. It’s easy to see why there’s interest in such a solution when it offers practical benefits without expensive, long, integration projects. I would suggest that for any sophisticated data access it is important to use a product that understands the context and processes of the data’s environment. Data always has some sort of context, and the value of data is not just degraded when context is lost; it could become inaccurate, irrelevant or misleading. A couple of examples for you to think about:
Temporal Context. Data source objects exist in different states as each system in the BSS/OSS stack operates on a different point on the line between Slow-Long-Term-Planning and Real-Time. For example, in an inventory a port may be seen as actively carrying traffic. On the NEM the port may be inactive due to a fault or planned maintenance. In a long term network planning tool the port might not even exist, instead being replaced by new equipment. All these data sources are accurate in their own context, but at any one time there may be a significant proportion of objects where their configuration, fault state, or existence differs between systems. An enterprise search application, trying to do the right thing, may start flagging up data discrepancies or breaking associations when in reality everything’s actually correct.
Process Context. Objects in BSS/OSS data sources tend to me manipulated through a trickle-down process. Resources are reserved by service fulfillment processes and recorded in inventory. Sometime later, hours or maybe days, the network is activated and services can be discovered through NEMs and faults tracked in fault management systems. This latency, as a change trickles through the systems, contributes to different temporal context. It also means data that could be flagged as related is not. Until the service delivery completes there may be no common identifier (IP address, MAC address, telephone number) tying the service, used port and used equipment together. That’s a shame, because it would be most useful at precisely that time to know that, for example, the target equipment (as set in the inventory) has a fault against it (as recorded in a trouble-ticket system) before the common identifier (an IP address) is activated on the device.
Context is not just an issue for enterprise search applications. Any system, whether enterprise search, federation, or synchronization, that joins data sources and also identifies discrepancies between them, needs to be aware of these ‘acceptable’ discrepancies and future associations. And it’s certainly not an impossible task. The application just needs to be aware of how the underlying systems operate, then react accordingly whether that be by not throwing a wobbly at a miss-match or by being a bit smarter about the way it builds associations between data sets.
A start-up like NexGenData has an opportunity to grab some of the wider enterprise search market, but they’re going to hit up against Google, SAP, and Microsoft who all offer generic enterprise search products. Merging search with other cool Web 2.0 features is a differentiator, as is a simple model for scalability. Even so, going for a specific market would be very wise. With a product that’s pre-tuned to how service providers operate, NexGenData would be able to offer total-cost-of-ownership below that of existing middleware while delivering business intelligence superior to the generic enterprise search vendors.
You can find out more about NexGenData enterprise data integration on their website, which is looking pretty good for an early stage start-up. There’s a good product overview, PR, data sheets and links to other industry commentators discussing this topic.
I’m not in a sceptical mood today, despite the grey skies, rain and hail. So as I write about vendor benchmarks you won’t see me using idioms such as ‘take with a pinch of salt’. I have no reason not to believe that OSS vendors' benchmarks, what few have been published, are anything but rigorous and representative of real-world use. I know for a fact that some of them are just that, with many man-months of effort to deliver useful data for internal development as well as PR-worthy headline figures.
How ‘representative’ a benchmark is to any telco other than the telco instigating or advising on the benchmark is up for debate. I talk a bit about the usefulness of benchmarks somewhere in here. Benchmarks are A Good Thing, for sure. They drive improvements to the product and encourage competitiveness between vendors. They demonstrate that the vendor has a system architecture which offers a good chance of concurrency and scalability. But they are not indicative of the performance any other telco will achieve with their different inventory network model, user profiles and business processes.
If you have an interest in an OSS vendor’s scalability and performance, then by all means look at their benchmark results. If you're interested in a hardware vendor’s server performance, then by all means look at their benchmark results too. But in both cases be very careful to pick out what is and is not relevant to your overall system design. There are likely to be fundamental differences between the benchmark design and your own requirements, although these differences are seldom obvious from reading the results published in PRs.
Perhaps the biggest gulf between benchmark and your own OSS performance lies in the evaluation of hardware and software platforms (OS, database and application server). As stated in previous posts, OSS system characteristics are very different from billing or CRM, but these BSS applications are often the ones a server vendor will use as the basis of their benchmark scenarios.
Just how different are OSS and BSS applications, in terms of system requirements? Amdocs recently put out a PR with IBM highlighting scalability of CRM on IBM servers. It didn't disclose a great deal of detail but the headline numbers illustrate my point. I'll compare its measurements with the only published Amdocs (actually, Cramer) OSS benchmark PR I could find: It's not exactly recent, but the size and scale of a tier-1 carrier doesn't change a huge amount six years.
What I want to look at is the requirements for the system, not the CPU/memory/disk-space that was actually needed (that's not published anyway). How do OSS and BSS requirements differ? Here’s the headline figures showing the scale of the tests.
Customers. 100 million and 60 million, which is somewhat more than the maximum number of customers most teir-1 division or subsidiary would currently need to manage. For example, the total population of Germany is ~82 million, France ~65 million, ATT Mobility has ~60m customers, T-Mobile US has ~25m customers.
User Count. BSS systems will have thousands of agent performing transactions throughout the day. OSS user counts are typically counted in hundreds. If there's a very large OSS user count it’s down to field agents/engineers who make only occasional use of the application. These benchmarks highlight a big difference in the quantity of users, but don't provide detail on the complexity of the work they are doing.
Interactions or Transactions. These are two different things. What constitutes an interaction or transaction is a later stage of discussion when sizing a system to support an OSS or BSS application. I’ve seen published Cramer OSS benchmarks with over 200,000, even 1 million, transactions per hour. Without getting in to a level of detail that’s not shown in the PR we will instead treat these figures as ’interesting’ but draw no conclusions today.
Database Size. The OSS database, for an equal number of customers, is larger than the BSS database due to the inclusion of detailed inventory and technical service information in addition to customer data.
At some point we have to sit down with everyone involved in system architecture design for the project and discuss user profiles, transactions and the complexity of inventory data. Before then, by using these high-level benchmark figures we can clearly highlight the differences between OSS and BSS, and justify the need for a deeper level of discussion.
I suspect I am preaching to the choir, and that as an OSS professional you're within your rights to accuse me of blogging the obvious. But many OSS customers out there come in to contact with other professionals who aren't as enlightened as you are. Many IT engineers, many server sales people and, dare I suggest, many system integrators do not fully appreciate how OSS platform requirements are so different from BSS.
By the way, if anyone knows of any other public benchmark figures from OSS and BSS vendors, I'd really appreciate being sent a link. Thanks!
