Thinkingcap technical issues white paper

Business Integration - a mirage or a worthwhile goal?

Business integration is much talked of in IT industry circles. It is rarely achieved in practice because of the poor integration of underlying IT systems.

In a survey by the Standish group, 88% of data integration projects were found to have failed. Despite this, business integration remains important to industry – a Morgan Stanley survey indicated that ‘business integration’ remains the top challenge and spending priority of CIOs.

What is business integration?

To achieve its business aims, an enterprise needs to act as a single integrated, responsive entity. Because all enterprises are now underpinned by IT, this implies the need for integration of the IT systems used. Furthermore, because specific business plans and goals change, IT applications and their means of integration need to be flexible. Thus ‘(enterprise) business integration’ implies a need for ‘flexibly integrated business IT’. This goal is not for IT departments alone but for the whole organisation. It can be labelled ‘the Holy Grail’ but organisations that have made good progress in building integrated and responsive IT systems have reaped rich rewards. One example is Amazon.

What is the problem?

Many business processes cut across traditional application and data store boundaries. A single business process will typically involve multiple applications and multiple updates to the data stores. This requires tying applications together with ‘glue code’ or, worse, manual data translation and transfer. Such a business process is inevitably fragile because the glue code has to be built bespoke.

The consequences of a fragile business process are a high error rate and poor control, leading to poor customer service. A fragile business process is not easily adapted to a change, be it a new customer channel, an increase in demand, or a new regulatory requirement. Adapting to such a change would either break the business process, often ruling out the change altogether, or stretch the business process so that it requires even more glue code and human support. This leaves it even more vulnerable to the next change.

Frequent external changes and aggressive near-term (6 to 12 month) business initiatives have exposed the rigidity of IT systems. The problem is so great that companies abandon initiatives rather than face changing their enterprise applications.

What are the pre-requisites for an integrated enterprise?

• A common enterprise-wide view of all its data, including the data’s structure and meaning.

  Ideally it would have only one record to refer to one real-world entity. It would not have, for example, duplicate (and potentially inconsistent) copies of a particular customer’s address. Every application would use the same source record, and furthermore would have a shared and unambiguous understanding of whether this was a business address, home address, billing address, shipping address or former address. An enterprise needs a common vocabulary.

• Enterprise wide business processes which depend on software functionality that uses the common vocabulary.

  The functionality can be incrementally changed. The impact of change in the data structure can be estimated. The functionality spans traditional application boundaries.  A traditional application could be as large a full ERP or CRM implementation or as small as a spreadsheet used by a single employee.

Do Service Oriented Architectures (SOAs) solve this problem?

No. They are a linking mechanism and do not provide the common vocabulary.

In return for the effort of SOA-enabling an application you get the ability to define software services that can be inspected and accessed by other applications. This is an important step for inter application communication. Most of the ‘technical friction’ of incompatible operating systems, languages and protocols is removed. Note the architecture within a single application is not affected. Changes that affect the fine-grained functionality of an application may not be any easier to implement.

The importance of a common vocabulary is largely ignored in the current industry debates involving service-oriented architectures (SOA). Interchange data formats or linking mechanisms cannot substitute for an agreed meaning.

What technology architecture is needed by an integrated enterprise?

An integrated enterprise should use a UES - a Unified Enterprise Server. This is a server process that exists to host all the business functionality and manage a common logical data store, i.e. one with an internally consistent data model.

Existing applications and their data stores should become components of the UES, or retired.  Also, they can start of as components and later be retired as the UES takes over the activities of supporting the business process. The UES can be replicated for performance and failover but, ideally, there is only one logical place where business functionality lives.

This diagram shows a UES, with an existing application, in action:

Is this too ambitious?

We believe not. It is, however, a long-term goal for a large corporation, though it should bring some intermediate pay-offs. It may well require some infrastructure investment, for example to support high rate data transfer.  A modern approach to server software development, like that pioneered at Thinkingcap, enables the creation of a UES. Data and application integration, data store unification and application retirement, are intermediate goals – the final goal should be the creation of a UES for the enterprise.

There are always likely to be disadvantages in trying to build a unified enterprise on the basis of separate applications integrated by a technological mechanism, whether that is a J2EE application server acting as an integration server, a message-driven infrastructure or a service-oriented architecture. This is the fundamental flaw in the ‘best of breed’ policies pursued by many organisations. True unification requires a common data model, a single logical data store and a server process that enables rapid change – true unification requires a UES. Once this is accepted, customers can start to define architectural principles that they want their suppliers to follow.

Creating a UES for an enterprise is potentially a long journey and one in which technology vendors must play their part. They will not necessarily do this willingly or quickly, as they have a whole set of business objectives such as revenue maximisation and customer lock-in that do not support their customer’s business aims. It is therefore crucial that, in transition, an enterprise receives value from a partially implemented UES that operates in concert with existing applications.

A UES based scenario

In this scenario a UES has been introduced to manage a new Customer entity. Customer has been created to tie together two existing entities that are managed by two existing applications:

The Customer entity starts off its existence to tie the other two together. This could be by one of:

• A common identifier across all three
• A reference in CustomerShipping and CustomerMarketing to Customer
• Two references in Customer, one to CustomerShipping and a second to CustomerMarketing. This would be the only option in the situation where the pre-existing applications do not allow additions to the entities they manage.

Once Customer is created as an entity managed by the UES, functionality that depends on both CustomerShipping and CustomerMarketing is now a possibility – for example, mailing out offers based on recent customer purchases and the customer’s profile. The UES would be a appropriate host for such functionality. It could directly access CustomerShipping and CustomerMarketing or go indirectly via the existing applications.

Adding a new Customer record is managed by a UES based application. Additions to CustomerShipping and CustomerMarketing have to be propagated to the existing applications or their data stores.

Often a new business process, needs new functionality and new data structures. If the enterprise wants to provide its customers with a loyalty card it will need to record more details for a customer, perhaps as more properties on the Customer entity or with a new CustomerLoyaltyCard entity or both. In this type of change there are significant implications for the UES:

• changes to existing entities
• new entities
• data migrated or added
• new functionality

Also, existing generic functionality in the UES (e.g. caching of data in the UES for better performance or a general report creation engine) must continue to operate in the face of the changes to entities and functionality.

This scenario illustrates some of the demands placed on a UES. It must:

• support the ‘reshaping’ of existing entities and functionality with the minimum possible impact and effort
• support the plugging in of new functionality with minimum impact
• provide generic functionality that is not dependent on the particular structure of entities or the business functionality
• support the addition of new generic functionality
• link to existing applications. This means support for a Service Oriented Architecture (SOA) as this is the current industry standard linking approach.

UES based integration

In planning a long-term goal it is important to identify how benefits will be harvested on the journey. We recommend an iterative approach that is data entity oriented. Each iteration focuses on adding to the UES the minimum number of logical data entities needed to support a new business initiative or internal rationalisation.

The first step is to perform an application audit. This involves identifying the applications in use and then classifying them in terms of the:

• the data entities it reads and writes
• the volumes of data involved and their rate of change
• the numbers of users and the types of usage
• its strategic value – for example, is it a large CRM application with in-house developed glue code around it or a small user defined spreadsheet
• the integration technologies it supports – native API, service oriented interface or direct data access

The aim is to get a measure of the effort required and a baseline ‘cost’ against which the benefits of a particular iteration can be assessed. The scope of the first iteration should be decided. Each iteration would comprise the following high level steps:

1. Identify those data entities that should be brought together in the UES. Good candidates are those entities that are currently managed by multiple applications and must be brought together to support a new business process.
2. Classify the applications into two categories: those which should be retired and those which need to be integrated into a UES.
3. Choose the integration approach for the retained applications. This could be to access their data store directly, use an application specific API or (preferably) use a service oriented wrapper.
4. Choose the retirement approach for the remaining applications. Issues to consider are the data conversion approach and support for the users of the retiring application. For example, a spreadsheet that holds master data could have its data mastered in a database and be generated as a read-only report.
5. Define any development projects needed to deliver the new business functionality.
6. Plan and execute the integration, retirement and development projects as a group.
7. Review the iteration and plan the scope of the next iteration.

In conclusion

We believe an enterprise should have a business integration strategy based on creating and leveraging a Unified Enterprise Server. The alternative laissez-faire approach of buying best of breed and using ad hoc integration tools or glue code will eventually clog up the arteries of corporations aspiring to be unified and agile.

There are a number of ways this objective can be achieved. One of these is to use Thinkingcap server as the basis of a Unified Enterprise Server. This is explored in a separate white paper ‘Using Thinkingcap as a Unified Enterprise Server’.