Anyone with a technical background will have a few pieces of
development work they're quietly proud of. It's not something I bring up at
parties to impress people, but I myself have fond memories of writing single
SQL statements that spread over two or three pages, incorporating those weird table-joins
whose syntax I can rarely remember and whose function is incomprehensible even
to most Oracle developers. This was back in early 1999 when I was working on
data migration for Cramer's second customer. It was my first practical lesson
in writing bulk data transform & data load that performed efficiently. Up
until then I had spent a few weeks writing code that worked correctly but
caused the office’s shared development server to thrash its CPU and discs. This
was before I knew what the database term 'full table scan' meant. The resulting
sluggish performance and my increasingly irate fellow developers forced a
review of my data migration strategy. Implementing some seriously hard-core SQL
was the answer at the time, but this could only be realised by thinking in
terms of telco network models, rather than assuming it was a simple job of
writing rows and foreign keys in to the database.
I am not going to get in to an analysis of data-load strategies right
now. Maybe later. Instead I want to ask
a question: Has anything really changed in data-load (or migration, or
transformation) practices in the last ten years? If you consider what most OSS
projects are actually doing today, I would suggest the answer is 'no'.
The concept of Extract-Transform-Load (ETL) applications has been
around for some time, and availability of ETL tools has increased, with most
middleware and database vendors offering some sort of 'value-added' tools on
top of an app-server or RDBMS. Are they being used by a significant number of
OSS projects? No. They're generic solutions and, as I mentioned at the start,
it is essential that the data-load process encapsulates network modelling
principals. I don't doubt that a good ETL tool could be configured for
network or service inventories and used successfully, but two things turn-off
project teams to this idea:
Learning
curve. Always wary of estimating project time for learning new tools, many
project managers would rather put in a proposal a bit of bespoke SQL or
Java development time. I'll bet most project managers already have their
formula: 'X device types * Y
service types = Z man-days'. Plus they can rely on a decent pool of people
to write SQL and Java, maybe even off-shoring it.- Performance.
Rightly or wrongly there's an assumption that generic data processing
tools are not tuned to the sort of through-put that is needed to migrate
tens of thousands of access points, millions of customers, and tens of
millions of circuits, in one long weekend (if your project schedule is
lucky enough to hit a long weekend for the switch-over). So, the projects
go for low-level SQL development where they at least have a chance of
producing something that performs well.
As a result, over the last ten years, off-the-shelf data migration
technology has had limited adoption in OSS, while in-house solutions have been
developed that offer little more than a framework and pattern for implementing
bespoke SQL data-load routines.
What is missing from the commercial ETL solutions is an application
design dedicated to supporting telco network structure and its data modelling
rules. With some out-of-the-box telco-specific capabilities the learning curve
is, at least partially, reduced and one can also assume the application is
built to offer acceptable performance when handling telco data.
Hoorah, then, for Celona.
The chaps at Celona have dared to invest in a product
that addresses the needs of OSS data migration projects. Assigning traditional
ETL to the 'second generation' and my beautifully crafted SQL being damned as
'first generation', Celona offer their 'third generation' business-centric
application data migration solution, Evolve.
Take a look at their web site and data sheets. They have created a
solution that goes a long way to address the two main barriers I mentioned
above. Beyond that, Celona has also had a go at solving the problems project
teams usually forget until they become critical. Unless you have done a big
data migration before, questions about how you swap over systems after (or even
during) the data migration are often asked too late in the project. Similarly,
questions about data cleansing may not come up until you find your first
conflicting database record.
Its early days for Celona Evolve. I would love to see some white
papers or customer testimonials demonstrating the effectiveness of the product.
I could then say things have changed since I cut those vintage SQL scripts ten
years ago.
I have my concerns though. Not from a technology point-of-view but
from the market’s. As I said, Celona's value proposition is based on solving
problems that I don't think every one of their potential customers is aware
of. The telcos are not data migration
experts. They pay IBM/Accenture/Amdocs/Whoever to worry about the detail. How
to convince them, then, that they need another line-item on their OSS product
license purchase order? Celona have a hill to climb in educating their target market.
Once Celona have potential customers accepting a problem exists and
their solution is the answer, how do they build an effective revenue stream?
'Mind your own business' I hear the Celona CEO saying. I normally would not
raise a question like this, but Celona's commercial proposition baffles me. Is
it a solution whose useful life is only as long as the first phase of the OSS
implementation project? How much can you charge for that? If the value proposition is a reduction in system integrator
man-days then that's nice, but will Cap/HP/Alcatel/Whoever be willing to
include Celona in the deal if it erodes their revenues?
Start-ups wouldn't be half as much fun if it wasn't for these sorts of
challenges. I went through the education and value-proposition thing in OSS
myself about 8 years ago. I think in 2000, maybe even 2001, I was explaining
what network inventory was to people who I would expect to know better.
It's great to see Celona have a well thought-out web site already, and
actively blogging on general data migration topics. They've got a lot of work
to do on educating the market, and I think they're on the right track with this.
I am not going to see a data sheet appear on Celona's website that
exposes their business model to satisfy my commercial questions, so I will
wait, eager and hopeful that a successful project press release appears
instead.
Whether you're looking for a commercial solution, or developing your
own data migration processes, I would recommend you check out Celona's website
and blog.