A Super- Platform for Data Integration and Business Process Services"

A Superplatform for Merger Driven Data Integration and Business Process Services

For those charged with "merger and migration marching orders”, OpenLink's Virtuoso Universal Server Provides an Ideal Data Integration Workbench and Web Services Infrastructure.

Alan Wilensky, Principal

Encompass Consulting, LLC.

"What goes around comes around. There is a major SOAP[1] opera playing out in the data access market place."

Kingsley Idehen, CEO, OpenLink Software, Inc.

Abstract

The swath of time straddling the late 20^th to early 21^st centuries might be coined, ‘the era of mergers’. Not only did AOL swallow Time-Warner whole, but hundreds of regional and neighborhood banks were devoured by behemoths like Bank of America. It is no longer a question of whether this is good for the consumer – it’s the default situation.

A dynamic of consolidation is the impact on technology procurement and human capital utilization. It is a major undertaking to integrate systems between merged parties. The Herculean task of wrangling the brittle business processes interposed between the merged entities is, to say the least, a vexing challenge – but one not without opportunities.

The savvy CIO being crushed between the stones of an impending or ongoing merger might ask, “How do new technologies, like Web Services, SOA[2], and ESB[3], make these merger gymnastics easier, more robust, and maintainable?”

In this monograph, the author will describe a new ‘superplatform’ for enabling the merger dynamic with the minimum of tool and platform splaying, and the ruminations of a great thinker in the enterprise data integration sector. This new superplatform is Virtuoso Universal Server, from OpenLink Software, and the profound thinking comes straight from OpenLink’s CEO, Mr. Kingsley Idehen, a true innovator in the database and enterprise middleware market for over 20 years.

The Merger Dynamic

Cultural issues aside, every merger implies the melding and reengineering of business processes. Behind these processes lies a vast sea of custom and commercial line-of–business software and supporting systems. Yet deeper we find a set of common data storage platform products, generally of the relational database ilk, that no modern enterprise platform can operate without.

This current state of affairs applies whether we are examining common ERP and CRM applications, or vertical applications for primary Line-of Business, such as banking. The core functions of these systems lie primarily within the underlying database stored procedures[4], and we can posit therefore that merging organizations and factoring business processes comes down to three key issues:

1) The ability to access data stored in the various critical Line-of-Business application databases,

2) The creation of application-to-application communication, decoupled from actual process overlap and dependencies, and

3) Process to Process communications that enable higher level, structured transactions with the current LOB applications within the enterprise, and with external partner systems outside the enterprise – some of which may be long running and demand intelligent failover and routing.

Later in this monograph, we will discuss the actual process of gaining access to and integrating these data sources, demystify the composition of inter-application communication using web services, and finally, create the composite business processes for creating new web-based applications from all of the foregoing, all while working with and extending the functionality of the current LOB applications.

Thoughts on the Process of Top-Down, Merger Driven, IT Systems Integration

There is tremendous post merger pressure to get systems working together, eliminate redundancies, and insure operational continuity; the first impulse, for even the most disciplined executives, are to let slip the dogs of IT, and watch the madness.

I asked Kingsley Idehen, OpenLink’s CEO, if there was a overarching, top-down philosophy, that could steer a merger right, technology aside? Kingsley was a top tier Unisys database integration expert, and single-handedly jump started the performance ODBC product sector – again, chief, the question, what’s the guiding principle?

Kingsley answers, “Each business has a set of processes that enable a number of critical success factors, and these are further decomposed when you drill down into the actual applications that compose the processes”.

So, a process might be the conjunction of a human in a seat, accessing several separate applications in order to complete one or more related processes, in order to accomplish a business task?

“You are correct”, says the chief, who is also one of the top SQL industry experts, “customer account management, inventory, POS - and then we hit the diverse data problem, the number one failure causing artifact in the IT merger game. This is caused by one or more systems, inherited from one side of the family or another, not clearly separating application logic from data storage, or not knowing precisely where the division occurs and can be exploited. But of course OpenLink has an answer in Virtuoso Universal Server”.

How can management drive the integration and insure a return on investment, and more importantly, a return on effort?

“Good question”, exclaims Idehen with a slap on the desk, “There are three points that must be tattooed on the foreheads of the merger team, both executive and IT management, as ‘merger marching orders’, to wit:

1) Consciously identify the market leadership discipline – innovation of cutting edge features and technology, operational excellence, or customer intimacy

2) Devise an IT plan, spell it out, as to how your leadership goal will leverage IT in the merger

3) Identify products that align with your chosen IT driven leadership roadmap – eliminate post merger systems that interfere, but most important, keep the underlying data and integrate!

…OpenLink has made this holistic process the critical foundation upon which the Virtuoso Universal Server is established, and it just happens to be the ideal environment in which to accomplish an IT driven merger”

“Think of Virtuoso not as a server infrastructure, or middleware platform, but as an encompassing environment that has all the tools needed to accomplish the complete spectrum of the integration process”, says Kingsley, continuing, “…but at the bottom of it all, is an enterprise class object-relational database engine and application server that leverages OpenLink’s twelve years in the universal data access market”.

OpenLink’s Product Spectrum Addresses the Merger Environment

OpenLink has a palette of solutions that satisfy the above lofty yet eminently achievable merger goals; The OpenLink Universal Data Access (UDA) product family is among the best performing and most secure ODBC connectors for the widest array of database management systems – the selfsame systems that house the aforementioned line-of-business data. Although the UDA family only allows a ‘one-to-one’ relationship between a customized application and LOB data – it is a valid portrayal of the separation of application logic from data storage.

In the Virtuoso product family, many data sources may be attached, and data from all LOB, legacy, and post merger sources, can be driven throughout the integrated environment.

Virtuoso performs with distinction in the realm of all competent enterprise databases, with the additional muscular ability to provide these integration services, perform bi-directional replications between all tables and indexes, and normalize SQL syntax across dissimilar data management systems.

Virtuoso has the additional benefit of providing an application services platform for all of this (now connected and integrated) post merger data. Data can traverse the Virtuoso server environment as XML generated from SQL data sources, and these dynamic XML documents can be made accessible to the world, or internally, via the ubiquitous URI[5]. Now that is short course in creating Web Services! Virtuoso may be the only data access and application server environment that allows even semi-skilled database administrative personnel to perform such integration and web services gymnastics in a no-code environment.

However, even after data integration issues are solved, the high-level merger team must face the challenge of domain and business unit process integration. On one side of the merger marriage, there may be ‘monolithic’ or ‘old-line’ business applications, where the processes are either not clearly documented, or must be retrofitted. Of course, in the case of recently created enterprise applications, such as the J2EE or the MS .NET family, the division of application logic and data may be a great deal clearer. We turn again to the OpenLink’s Chief, in order to get a handle on how to proceed:

”The problem here is that such an effort requires this re-factoring of application functions. We now have the data where we want it – we now have to re-design by isolating functions, and this is the heart of Web Services. We will take the pre-merger entities, and decompose each LOB function on both sides into component services. Then, we can eliminate redundant function, and expose the keepers via SOAP (Simple Object Access Protocol), and WSDL (Web Services Description Language), all with security and reliability based on W3C standards”, says Kingsley in pleasant, British phonemes.

Virtuoso, as it happens, does this very well, as it is a native run-time environment for not only connected and internal SQL stored procedures, but for the whole panoply of application languages, i.e., Java, .NET, C++, Python, and PHP.

One of the reasons that OpenLink can call Virtuoso a “Universal Server”, with a completely straight face, is that Virtuoso can run a Microsoft .NET language run-time on a Linux Server; making this is an encompassing data integration and application serving workbench worthy of its name.

The merger team rounds third base by creating the composite business services, or more properly, processes that are the end-product of our services decomposition.

Some process must run for a relatively long time… certainly longer than a single user session, as in a loan applications process. This also occurs in retail and manufacturing, where trade exchanges for inventory replenishment and cost optimization is a relatively new phenomenon. Note that we say relatively – older EDI[6] (Electronic Data Interchange), POS, and banking are still carried over a dwindling yet still costly handful of value added networks (VANS).

Kingsley expounds further on this topic, ‘The advent of the XML era and Web Services – the Service Oriented Architecture so touted, has rendered most EDI and VAN applications obsolete. Why? In the old days, while I was at Unisys, all that trading-partner data transit came from relational data sources. Where in the world was all of this XML going to come from? This is why so many small and innovative pure XML database vendors failed or were bought cheap.”

Continuing, “Now, with twelve years under our belt, in the UDA business, and Virtuoso entering its 4.0 version, we cover the full-cycle of relational data to XML to Web Services, back to relational or object data. With Virtuoso’s Business Process Execution Language engine, we cover macro business processes, reliable transactions, and external trading partner transactions”.

I wanted a bit more from Kingsley, so I asked where these layered transactional services would be applicable. “For instance, OpenLink will be working with Rosettanet.org to enable XML based trading between companies in electronic component distribution and semiconductor manufacturing. Similarly, Virtuoso will work with other XML schemas, such as the ACORD[7] schema commonly employed in the banking and underwriting industries”.

Synopsis

The merger dynamic implies a gargantuan task spanning organizational cultures and technology. The choices today are more than just numerous – they are confounding. Amplifying the noise within the Web Applications solution space is an IT media juggernaut that cedes editorial coverage to major industry players and the professional[8] analyst trade.

It is refreshing to examine a product that is the brainchild of a single, bright and focused mind – one of imagination and bedrock practicality. Such a product is Virtuoso Universal Server; a complete environment for gathering, integrating, and deploying new services across the deep reaches of an organization’s entrenched systems, and spanning the gulf that exists between new partnerships.

Not only is Virtuoso a workbench for the merger dynamic, but it is truly a Superplatform for re-enabling and invigorating completely new services and organizational processes when paired with contemporary line-of-business applications, from whatever industry.

Virtuoso stacks up well as a unified data access system, application hosting environment, and business process orchestration environment. If your team has received ‘IT Systems Merger Marching Orders’, it’s good to remember not only the virtues of OpenLink’s Virtuoso, but the pedigree of the company, it’s founder, and the incredible team they have assembled.

A Primer of Line of Business Applications and the Relationship to Underlying Database Management Systems

In the few months that I have been working with Kingsley, I have learned a great deal about application integration and deploying merged services across corporate boundaries – I have also had many myths dispelled.

Q: When you deploy any of these enterprise solutions - it’s understood that there is a big three database product sitting underneath before anything will work, i.e., if it's a new install, here comes a SQL Server or Oracle, or DB2 license, comparable in cost to the ERP package - or if you have been running some flavor of database as an anchor application (as many do), then you are in for an upgrade that’s going to hurt. Can any of these fat LOB apps be retrofitted to use an Open Source or Alternative Database Engine, such as Virtuoso’s core engine?

A: Spot On. LOBs (JDE, SAP, PeopleSoft etc) are typically built atop Oracle, DB2, and SQL Server. Rarely are they built atop ODBC or JDBC, and never specifically atop MySQL or PostgreSQL[9].

Web based applications on the other hand are typically built atop MySQL (native interface), PostgreSQL (native interface), and SQL Server (via ODBC under Windows). Note that native interfaces are called Call Level Interfaces (CLIs) in DBMS parlance. These CLIs are very ODBC-like (OpenLink has a kit for converting MySQL CLIs based apps into their ODBC equivalents).

LOB vendors are DBMS specific as a result of development myopia. They typically claim that DBMS independence isn't attainable, hence the DBMS specificity of their products. They aren't deliberately excluding MySQL or any other DBMS, the issue is DBMS specificity at the data access level. SAP also had their own SQL DBMS Engine which they sold to MySQL sometime in 2003.

The data access level is how data enters a DBMS and includes the routines for managing the various data manipulations. A LOB that isn't ODBC or JDBC based cannot prevent data access via ODBC or JDBC for instance. Thus, you can access data created by LOBs using ODBC, JDBC, OLE DB, ADO.NET etc. You can interrogate the schemas of these DBSM engines and unravel the data structures and persistent Stored Procedures using ODBC, JDBC, OLE DB, ADO.NET etc..

SAP or any other LOB can leverage VUS to be genuinely DBMS independent for sure. Most are unaware of the existence of such technology.

Q: Is there a way for common LOB/ERP packages to natively provide data integration services - in the case of products preceding Virtuoso Universal Server - common database middleware - brute force API work, mass batch data migration? We are discussing enterprises that have large datasets and historical or recent transaction data; I fail to see how SAP et al could have ignored that requirement?

A: They have ignored this requirement because they strategically seek to become "one stop shops". At best, they may have ODBC and/or JDBC access via OEM or reseller relationships with middleware vendors. You may be aware that PeopleSoft (now owned by Oracle) made a $60M acquisition of Cohera from Dr. Mike Stonebraker. (Founder and CTO of Cohera). They never put this data integration to any use.

Q: We know that many manufacturing, logistics, and telecom leviathans ran on VAX VMS, IBM VM and System OS390 - AS400 - as well as older ISAM and VSAM libraries, and older Oracle, MS SQL, etc., - are these all historically being managed by industry specific System's Integration specialists for data integration?

A: Horizontal and vertically focused SIs (some of these are VARs that have morphed into SIs).

Q: What about the usual suspects of TIBCO, BEA, Web Methods – it is a crowded market with newcomers such as IONA, Cape Clear.

A: They all need data integration and data access. Today, they may go to a variety of data access tools vendors, but they are perhaps missing an important opportunity regarding heterogeneous data integration and data access of the form delivered by Virtuoso (or the security sensitive data access of our UDA product family).

Hopefully, this paper and OpenLink’s renewed outreach will better broadcast that message….how much am I paying you?

Q, Does a company such as SAP or Seibel need to have some type of internal data integration suite?

For Example: Say that I own a sheet metal fabrication company, and that I run a legacy vertical LOB application, and a combination of AS400 for my inventory system, and a PC server for invoicing and a plug-in for some accounting system such as Great Plains. Now, I want to go SAP; where do I start?

A: This is where Virtuoso would come into play; how do you get your data into SAP? This is a Classic data access challenge. SAP may not have the complete repertoire of tools for easy migration. The bigger opportunity lies in the accessibility to SAP via its Netweaver APIs which are SOAP based. BPEL is about constructing composite processes from a myriad of discrete Web Services.

Q: OpenLink claims that application functions can be hosted and run cross-platform- - can you explain the data access methods for these LOB applications? Are there accessible frameworks in which to break out functions for merger driven integrations?

A: These LOB applications implement their application logic using SQL Stored Procedures. That's how you write SQL based LOB applications. In rare cases this may be locked via 4GL style code, but this is rare (in certain cases 4GL code generates libraries that we can exploit). However, a majority of the code in question is implemented as SQL Stored Procedures.

Q: Most custom applications are written in Java, C++, or more recently, C#, and complied as specific binaries, or a CLR, and then linked into executable packages. Are you saying this is not the case for SAP or Peoplesoft?

A: The functional core of these popular LOB applications is SQL Stored procedures. Architecturally, there are many tiers to these applications; presentation, application / business logic, and data storage. The application is typically Stored Procedure based. In the case of C/C++ style libraries, you can integrate these with Virtuoso once one understands the interfaces that are implemented. These bindings are at the API level. We are talking about interfaces and implementations which may take various forms such as Java assemblies, C/C++ modules (functions and objects respectively), accessible via dynamic / shared libraries, and .NET assemblies.

The constant is the data repository. In most cases we can take the data into the Virtuoso space (in a variety of ways) and perform the decomposition there.

Q: In a Bank merger, using a costly integrated banking application, or a big SAP deployment (which I have to assume is somewhat more open), how do you decompose functions for deployment via web services?

A: We locate the SQL Stored Procedures, Java Classes, .NET assemblies, or C/C++ modules hosting the application logic on a functionality realm basis. In the worst case we use Virtuoso to create wrapper stored procedures that create equivalent functionality abstractions pre exposure for Web Services protocol invocation.

Note, this process may require hand holding in many cases, but this is a small price to pay in relation to the benefits of the ultimate solution. Each vendor worth its salt provides APIs (yes these take the form of DLLs, Shared Libraries, Java Classes, .NET Assemblies). Beneath these APIs lie Stored Procedures and we can build connectors into Virtuoso that aren't bound to J2EE.[10]

Q: Kingsley, I hear you are quite the database historian, and have been around for some of the key innovations; can you give us a peek at your history?

A: I worked on SequeLink before I founded OpenLink; I assume you see the connection to the company name. In the beginning there was Pioneer (what is today Data Direct), TechGnosis (SequeLink), VisionWare, and a consulting firm called PAL Consulting (now OpenLink Software).

Today, VisionWare is out of the market (part of Tarantella Inc. nee the real Santa Cruz Operation (SCO as opposed to SCO Group nee Caldera)), TechGnosis and Pioneer (after becoming Intersolv, and Merant) are now part of the Progress Software unit called DataDirect.

When we were PAL consulting, I worked on a product called SequeLink for Progress. In short I was one of the first people to have Progress exposed to data access middleware. What goes around comes around!

Indeed!

[1] This is the author’s play on words and nomenclature – Mr. Idehen meant soap opera in the colloquial sense, while the author re-cast the quote as a play on the Simple Object Access Protocol, a foundational Web services technology

[3] Enterprise Service Bus – a figment of the industry’s imagination

[4] Stored procedures are declarative programs that run in response to a database operation – usually written in SQL or a vendor specific language

[5] Uniform Resource Identifier - A new name for URL

[6] EDI is a mature text encoding and format standard for purchase orders, and wire bound files transmitted over private networks – the sun is rapidly setting on EDI and transitioning the EDI vocabulary to XML data.

[7] Association for Cooperative Operations Research and Development)

[8] Professional = Paid to cover a company or product and never objective

[9] The two top Open Source relational databases