OSS Line

Celona Gets a Wodge of Cash

Celona, one of the more interesting start-ups out there, has announced a tidy £2 million investment from VCs firm Seraphim Capital and existing investors.

Over the last year Celona has started to demonstrate progress in winning business and deploying their data & process migration solution at telcos like Telstra and Swisscom. Along with getting in to the Red Herring Global 100, this further investment in Celona suggests that the industry is starting to appreciate their unique approach to dealing with a problem that is usually ignored then subsequently bodged by throwing professional-services bodies at the problem.

Seraphim already have skin in the OSS game with their existing investment in planning and optimization start-up (and employer of yours-truly) Aria Networks.

More information can be found in the Celona and Seraphim PR.

TMW Nice 2009, What's Hot and What's Not

HOT! Optimism

There's more of a buzz than last year. After the peak in M&A in 2007/2008 last year felt like a career fair with many attendees being more focused on re-evaluating their careers. This year, people are milling around the expo, enjoying the catalyst projects, and talking about technology and business. Maybe its because there's no jobs out there....

NOT! Booth-Bunnies

It's not exactly the Motor Show, but there has been an increase in show budget being spent on eye-candy for the predominantly male attendees. Maybe the most affordable staff to hand out product brochures happen to be young attractive girls? I will give them the benefit of the doubt. But models aimlessly stood around just looking pretty... a bit tacky.

HOT! World-Weary Pessimism

To paraphrase two commentators this week: "It seems like TMF is doing the same thing they were doing three years ago, and Google, Amazon e.t.c. have moved things on without them". "We're sitting here discussing the same things we tried and failed at already, and were going to fail at it again before we get it right". Say it like it is...

NOT! No New OSS Start-Ups Here

I know they're out there in the real world, and I know budgets are tight, but it would have been nice to see some of the industry start-ups manning their first tiny stand at the far back-corner of the expo. Next year maybe?

HOT! Start-Up Success

Ontology and Celona are coming along nicely. There's no Cramer-style late-90's growth, but there is steady progress and evidence that they are creating new markets for themselves.

NOT! Give-Aways

Not so much that there weren't any, but the bounty I received from Amdocs (memory key, umbrella, plastic thing, note pad, torch) has left me with no room in my bag for anything else. Any of you vendors want to post me some of your left over prezzies?

HOT! Free Drinks

Thanks to all the companies that supplied their own end-of-day refreshments. Great for those of us with cheap TMW passes that don't include 'cocktail reception' tickets. I particularly appreciated Convergys' chilled beers in their own little stay-cool vest.

Data Migration - Amdocs and Celona

A couple of months ago I talked about data migration and how Celona is taking a ‘productised’ approach to solving the issues that are usually addressed by bespoke integration. Yesterday, Celona teamed up with Amdocs to deliver a webcast on data migration approaches for network inventory. I was keen to find out what’s new with Celona, and learn a bit more about the nature of their publicised relationship with Amdocs.

If TMF post up a recording on their site I suggest you take a look. As webcasts go it was quiet informative, if you have any interest in data migration issues. It explains very clearly Celona’s approach to what is traditionally an area of the system architecture that is rarely designed well or implemented effectively.

Three key points that made me go ‘ooooh’ and/or ‘hmmm’ were:

Celona provide real-time migration. Unlike traditional systems there's no intermediate database or staging of data before loading. Celona performs real-time in-memory manipulation of the data streams. The key benefit of this is that you can have a legacy and target system both running at the same time on a shared dataset, with Celona doing bi-directional updates. Nice!

Celona’s business model. In my previous blog entry on Celona I ask how they generate reliable revenue streams given the perceived short-termness of migration projects. Someone (not me!) asked the direct question. Tony Sceales (CTO) said that they have a flexible approach selling direct or through partners. He made an interesting point that, usually, there’s an on-going need for migration in the customer’s organization. Once a customer looks at the problems and Celona’s solution, he expects them to want to take their product license and use it on future projects.  For partners, Celona support the concept of ‘migration factories’, where the SI buys the product license and uses it as a basis for selling some sort of value-add migration service. Hmmm... I am still not convinced they’ve provided a conclusive answer to the question. 

Amdocs-Celona relationship.  Celona are part of Amdoc’s partner programme, but it wasn’t something they went in to in much detail. Reselling? Amdocs delivering Celona products? Preferred product status? Who knows. The meat of the relationship is the pre-built adapter that allows Celona to integrate with Amdocs Resource Manager inventory. The only pre-built adapter to a COTS product they’re marketing, as far as I can tell. Dipak Pankhania (Celona Senior Consultant) was keen to stress that they intend to keep this adapter up-to-date as new Resource Manager versions are released.

Celona teased the audience with stories of their on-going project at a major service provider. Naming no names though. Hopefully we’ll soon see some detail about a commercial and project success.

OSS and Data Migration Scalability - It's Not Just About CPUs

I proudly discussed my finely crafted SQL statements in a previous post about Celona and data migration. The need to invest quite some time in learning a few more SQL tricks and building logically complex and very large single SQL commands (as opposed to conceptually simpler procedural PL/SQL routines) came about because I was trying to migrate some data in to a network inventory (Cramer Dimension 1.0 I believe). Without this approach I found that through-put was terrible, perhaps as slow as a couple of objects a second once the data tables grew past a certain threshold. And no one else could use the shared server, despite it's CPU utilization being pretty low at the time. It turned out to be an early lesson in Oracle full table scans. As the name suggests, almost every look-up I was doing (to 'join in' the new object to the rest of the network) resulted in Oracle having to scan through every single row in several tables. It wasn't CPU intensive, but it was thrashing the server's hard disk. The process was 'I/O bound', limited by the performance of storage resources, due to having to process all that data for every object creation.

I admit fault. It wasn't a problem with Oracle. My code, PL/SQL which I can now accept was naively written, was making Oracle's job harder than it had to be. The second lesson was that if you told Oracle what to do in a concise but precise way using SQL, rather than try and spell it out in procedural code, you'd often get far better performance.

I've seen similar issues many time since in my own processes (less so, I like to think, as time goes on) and in others. The code is programatically and syntactically correct. There's plenty of CPU power. Lots of RAM. The performance still sucks. This tends to happen a lot in data migration, transformation and load projects, probably because they tend to lead to people crafting new code and doing so isolated from the large data-sets it will eventually have to cope with.

It does seem to baffle people that its not about CPU power. That their 3Ghz quad-core workstation or 12 CPU server can only process two objects per second would almost be amusing except for the project plan re-factoring needed to accommodate a go-live swap-over with 1.5 years of planned down-time.

So what's the problem? "James", you're saying, "tell us which SQL function or Oracle config file we should be looking at".

It's unlikely to be that simple. If you've hit a problem like this there's a good chance that the overall architecture of the process, whether that be an OSS provision process or data migration process, is fundamentally wrong. If you're very lucky, and didn't hit problems from the outset, you might just have one or two dodgy sub-routines causing the problem.  In either case the possible root causes are many and varied.

It would be a long blog post if I detailed even a few architecture 'anti-patterns', database design best practices, or optimal SQL statements. Instead, let me illustrate both the scale of the issue, and the therefore areas you might want to start looking in to. Here's the big three issues that lead to poor scalability...

Lack of concurrency by process design. You have a process that operates sequentially, handling one object at a time. This may perform reasonably well on a single CPU, single disk desk-top PC, but it will not scale up to use multiple processors or make the most of enterprise-class storage arrays. Consider designing the process so that you can run multiple instances on batches of data, but be very careful of data integrity issues that must then be dealt with.

Lack of concurrency by data model design. You have concurrent processes running, but they're becoming serialized due to limited, shared database resources being locked. Any time you explicitly lock data or implicitly lock it as part of a data modification you will reduce concurrency. Locking is essential for data integrity but too much locking where it's not needed will result in serialized processes. That means one process, in effect, and therefore only one of your server's CPU cores having any real work to do.  Find the bottle-neck in the data-model and carefully look at how processes lock that resource.

Poor execution of database queries. We know the point of relational databases is that they do lots of clever stuff under the hood/bonnet to optimize how data is extracted for any given query. But we have to accept some responsibility for performance and ask queries in an appropriate, precise and concise way. A single badly written SQL query could take minutes to return results, and thrash the disks limiting any other work on the server. The same query, re-written, could return the same result in under a second with minimal I/O resources used. It's all down to the structure of the SQL, and can also be influenced by 'indexes' in the database that help with specific queries. There's about half a dozen common and fairly obvious (if you know what they are) causes of poor SQL performance, in Oracle specifically.

Any process that has to deal with large amounts of data, which is most OSS and data migration tasks, has to be designed with performance, scalability and concurrency in mind. The most important thing is to know your database intimately. Grab a good book, specific to Oracle, mySQL or SQL Server, whatever you're using. Pick up one on SQL tuning or database architectures and read the first three or four chapters at least. Most authors will cover the simple things that make the biggest different to scalability early on (ever read a Tom Kyte book without being told about bind variables in the first three pages?). Use them. In fact, write key code patterns in to your design documents.

Beyond technical knowledge of the database, a deep understanding of network inventory is also needed. What are the options for modeling, say, MTOSI-base physical objects? What data needs indexing to answer the sort of questions fired at an inventory? How should processes apply concurrency and data-integrity rules when modifying the network model in the database?

That's something I'll write about another day.

Data Migration and Celona

Anyone with a technical background will have a few pieces of development work they're quietly proud of. It's not something I bring up at parties to impress people, but I myself have fond memories of writing single SQL statements that spread over two or three pages, incorporating those weird table-joins whose syntax I can rarely remember and whose function is incomprehensible even to most Oracle developers. This was back in early 1999 when I was working on data migration for Cramer's second customer. It was my first practical lesson in writing bulk data transform & data load that performed efficiently. Up until then I had spent a few weeks writing code that worked correctly but caused the office’s shared development server to thrash its CPU and discs. This was before I knew what the database term 'full table scan' meant. The resulting sluggish performance and my increasingly irate fellow developers forced a review of my data migration strategy. Implementing some seriously hard-core SQL was the answer at the time, but this could only be realised by thinking in terms of telco network models, rather than assuming it was a simple job of writing rows and foreign keys in to the database.

I am not going to get in to an analysis of data-load strategies right now. Maybe later.  Instead I want to ask a question: Has anything really changed in data-load (or migration, or transformation) practices in the last ten years? If you consider what most OSS projects are actually doing today, I would suggest the answer is 'no'.

The concept of Extract-Transform-Load (ETL) applications has been around for some time, and availability of ETL tools has increased, with most middleware and database vendors offering some sort of 'value-added' tools on top of an app-server or RDBMS. Are they being used by a significant number of OSS projects? No. They're generic solutions and, as I mentioned at the start, it is essential that the data-load process encapsulates network modelling principals. I don't doubt that a good ETL tool could be configured for network or service inventories and used successfully, but two things turn-off project teams to this idea:

1. Learning curve. Always wary of estimating project time for learning new tools, many project managers would rather put in a proposal a bit of bespoke SQL or Java development time. I'll bet most project managers already have their formula:  'X device types * Y service types = Z man-days'. Plus they can rely on a decent pool of people to write SQL and Java, maybe even off-shoring it.

2. Performance. Rightly or wrongly there's an assumption that generic data processing tools are not tuned to the sort of through-put that is needed to migrate tens of thousands of access points, millions of customers, and tens of millions of circuits, in one long weekend (if your project schedule is lucky enough to hit a long weekend for the switch-over). So, the projects go for low-level SQL development where they at least have a chance of producing something that performs well.
   

As a result, over the last ten years, off-the-shelf data migration technology has had limited adoption in OSS, while in-house solutions have been developed that offer little more than a framework and pattern for implementing bespoke SQL data-load routines.

What is missing from the commercial ETL solutions is an application design dedicated to supporting telco network structure and its data modelling rules. With some out-of-the-box telco-specific capabilities the learning curve is, at least partially, reduced and one can also assume the application is built to offer acceptable performance when handling telco data.

Hoorah, then, for Celona.

The chaps at Celona have dared to invest in a product that addresses the needs of OSS data migration projects. Assigning traditional ETL to the 'second generation' and my beautifully crafted SQL being damned as 'first generation', Celona offer their 'third generation' business-centric application data migration solution, Evolve.

Take a look at their web site and data sheets. They have created a solution that goes a long way to address the two main barriers I mentioned above. Beyond that, Celona has also had a go at solving the problems project teams usually forget until they become critical. Unless you have done a big data migration before, questions about how you swap over systems after (or even during) the data migration are often asked too late in the project. Similarly, questions about data cleansing may not come up until you find your first conflicting database record.

Its early days for Celona Evolve. I would love to see some white papers or customer testimonials demonstrating the effectiveness of the product. I could then say things have changed since I cut those vintage SQL scripts ten years ago.

I have my concerns though. Not from a technology point-of-view but from the market’s. As I said, Celona's value proposition is based on solving problems that I don't think every one of their potential customers is aware of.  The telcos are not data migration experts. They pay IBM/Accenture/Amdocs/Whoever to worry about the detail. How to convince them, then, that they need another line-item on their OSS product license purchase order? Celona have a hill to climb in educating their target market.

Once Celona have potential customers accepting a problem exists and their solution is the answer, how do they build an effective revenue stream? 'Mind your own business' I hear the Celona CEO saying. I normally would not raise a question like this, but Celona's commercial proposition baffles me. Is it a solution whose useful life is only as long as the first phase of the OSS implementation project? How much can you charge for that? If the value proposition is a reduction in system integrator man-days then that's nice, but will Cap/HP/Alcatel/Whoever be willing to include Celona in the deal if it erodes their revenues?

Start-ups wouldn't be half as much fun if it wasn't for these sorts of challenges. I went through the education and value-proposition thing in OSS myself about 8 years ago. I think in 2000, maybe even 2001, I was explaining what network inventory was to people who I would expect to know better.

It's great to see Celona have a well thought-out web site already, and actively blogging on general data migration topics. They've got a lot of work to do on educating the market, and I think they're on the right track with this.

I am not going to see a data sheet appear on Celona's website that exposes their business model to satisfy my commercial questions, so I will wait, eager and hopeful that a successful project press release appears instead.

Whether you're looking for a commercial solution, or developing your own data migration processes, I would recommend you check out Celona's website and blog.