Enterprise Application Integration - Where Is It Now (And What Is It Now)? Part 1: What Is It Now?

Part 1: What Is It Now?
M. Reed - September 3, 2001

Summary

Since January 2000 when TEC last addressed the trends in Enterprise Application, there have been massive changes in the overall direction of Application Integration in general and EAI in particular.

A great many of the players have changed in the vendor arena, new terminology ("buzz-phrases" like IAI, or "Inter-Enterprise Application Integration", and "e-Business Infrastructure Enablement", or B2Bi), has arisen out of the marketing machine, and customers are generally confused as to just what vendors are offering and how it may (or may not) solve their business problem. In addition, the strong downturn in the technology economy has affected many vendors' prospects.

The definition of EAI is amorphous at best, so TEC has sought to take the broadest definition, which is as follows:

• Enterprise Application Integration is any process, or series of processes, which enables business capabilities through the combination of data and business logic obtained from separate systems. Whether the data is internal, inter-divisional, customer or supplier facing is irrelevant. Vendors can give it whatever term they choose, or sub-divide it into separate applications, but to us, integration is integration. Plain and simple. IT programmers have been doing it for over 20 years.

This is a two-part note on EAI trends. This part discusses the basic components to achieve EAI. Part Two covers the Market Impact of the recent changes and makes User Recommendations.

EAI Basic Components

The basic components required to achieve EAI are the following:

Business Rule Component: to allow the applications to understand your business processes.

• Business Logic Modules (i.e., supply planning, sales order processing. Methods for business process management). This is perhaps the most important but least understood aspect.
  
  
• Transformation tools (to define how to map data from one system to another)

Data Acquisition Component: to allow access to the data.

• Data Source and Target Interfaces (i.e., Siebel, SAP, PeopleSoft, ODBC, JDBC, Oracle, CICS, IMS, IBM DB2 UDB). Note that the data acquisition component is crucial to EAI success, and is tied to Data Cleansing (see below). Most vendors refer to these interfaces as "adapters" or "connectors". Especially to many large corporations, efficient access to legacy systems such as ADABAS or IDMS may be required, and not via "flat file extracts" which are expensive, time consuming, and bandwidth intensive. In addition, Event-based data acquisition can be difficult to define, usually requiring database triggers or database log analyzers. When the data source is not a relational database (i.e., a text file), this process can become extremely complicated.

Data Cleansing Component: to ensure the data is correct.

• Few things lead to bad business decisions faster than dirty data (for more information see "Business Basics: Unscrubbed Data Is Poisonous Data").

System Development Component: to allow programmers to design and test custom requirements.

• Design tools (for business process design, data mapping and transformation design, debugging, and testing)
  
  
• A published API (Application Programming Interface) to the product so custom extensions can be written as needed (and they are always needed)

System Control Component: to allow the application to be monitored and controlled.

• Management tools (for application-specific monitoring), preferably with support for the Simple Network Management Protocol (SNMP) via a vendor-supplied SNMP agent. TEC considers this the most important component currently lacking in the major system management products (IBM Tivoli TME, BMC Patrol, and Computer Associates Unicenter). All three vendors have indicated that they understand the need to help their customers track distributed units of work, especially in heterogeneous EAI environments, but all admit they are not quite there yet. It will be a daunting task indeed. The IT manager must be able to know exactly where any process is at the time, similar to the way JCL batch jobs on mainframes were monitored in the past. Questions such as what step is the job in, how much CPU time has it absorbed, etc. must be answered in real time.
  
  
• Commitment control mechanisms (for control of business-level logical units of work) - These mechanisms allow the systems involved to recover from failure without corrupting the data in either system. This becomes much more difficult when more than one type of database or file system is involved in the logical unit of work.
  
  
• Strong support for metadata management, preferably distributed and bi-directionally synchronized. This point also can not be emphasized enough. Without distributed metadata management, IT personnel continually have to reinvent the wheel, leading to errors and misleading results. Metadata repositories from vendors such as Viasoft and Computer Associates call be helpful in this regard.
  
  
• Message Brokers (to control transactions, control security, and perform event notification, e.g.,.IBM's MQSeries, or Microsoft's MSMQ for Windows NT/2000) - The product should also include the capability to "bridge" messages between different messaging systems (e.g. an IBM MQSeries mainframe application that needs to communicate with a Microsoft MSMQ application on a Windows platform).
  
  
• Scalability for high-volume transaction throughput - We cannot put enough emphasis on attention to scalability. It is almost impossible to know at implementation time what the data volumes will be in the future.
  
  
• Support for varying levels of fault tolerance, load balancing, and failover for mission-critical systems
  
  
• Workflow enablement (via SMTP, publish/subscribe capabilities, etc.) is a key requirement to reduce latency between distributed processes. The product should also have links to workgroup products such as Lotus Notes Domino and Microsoft Exchange.

System Access Component

• A Web-based portal for transparent, integrated information access
  
  
• Many vendors overemphasize the importance of the portal. We believe that, although the access method must be highly useable, the underlying technologies are most important. A good example of a vendor overlooking things is Computer Associates (see, Computer Associates Jasmine ii: When is a Portal Not Just a Portal?). The vendor has admitted that they may have underemphasized the importance of the infrastructure in favor of the portal, at least temporarily.
  
  
• The ability to work both inside and outside the corporate firewall (for instance using HTTP sockets).
  
  
• Web Application Servers (generally using Java 2 Enterprise Edition, or J2EE), also fit loosely into this category.

Understanding Data-oriented Middleware

To attempt to explain EAI, an understanding of data-oriented middleware is necessary. In addition, the "portal" concept must be explained (see glossary), as the web is the standard front-end to these distributed technologies. The topology of this type of application can be split into two parts; the first is the middleware that connects the disparate data sources, the second is the business logic that provides a unified view of the data. Given the complexity of this solution, consulting is required to integrate all the pieces, as there is no "out of the box" solution that can provide plug-and-play integration.

Coupling Time and Process

Much consideration must be given to the coupling of time and process. There are three basic communication types which may be required for the middleware solution:

• Asynchronous: provided by products such as IBM's MQ Series, used in situations where the communicating applications don't need to be active at the same time. This allows applications to be "loosely coupled in time". An example of this would be a terminal application feeding a batch process.
  
  
• Synchronous: provided by OMG's CORBA, Microsoft's COM, and the new emerging standard SOAP (Simple Object Access Protocol, developed by Microsoft but now being embraced by the industry since IBM now supports it), used where the business logic requires that the applications communicate in real-time. Generally the requesting transaction waits until it receives the result set from the other application.
  
  
• Transactional: using transaction monitors such as IBM's CICS or BEA's Tuxedo. This is required where the logical unit of work (defined as one business transaction) spans multiple systems.

Security

The last major requirement for an EAI system is security, typically in the form of single sign-on (to provide access to all the required data without multiple passwords.) Using EAI provides a business with the ability to consolidate "stovepipes" of information from various transactional, legacy, relational, and other sources with different security rules and requirements into one unified and secure view. Security becomes even more crucial when companies allow access to internal data by outside sources.

Rationalizing the Data

An additional challenge is the need to rationalize the data in the different data sources. For example, the definition and datatype of the "customer" field may vary from one database to another. These dimensions need to be conformed in order for any meaningful integration to take place

This completes Part One of a two-part note on the Trends in EAI. Part Two discusses the Market Impact of recent changes and makes User Recommendations.

Glossary

ODBC: Open Database Connectivity. A database programming interface from Microsoft that provides a common language for Windows applications to access databases on a network. ODBC is made up of the function calls programmers write into their applications and the ODBC drivers themselves.

JDBC: Java Database Connectivity. A programming interface that lets Java applications access a database via the SQL language. Since Java interpreters (Java Virtual Machines) are available for all major client platforms, this allows a platform-independent database application to be written. JDBC is the Java counterpart of Microsoft's ODBC. Java was originally developed by Sun Microsystems.

OLE DB: OLE (Object Linking and Embedding) Database. A programming interface for data access from Microsoft. It functions in a similar manner as ODBC, but for every type of data source not just SQL databases. Applications can use OLE DB to access ODBC databases as well. OLE DB for OLAP is used to access OLAP databases. OLE DB is a COM object.

COM: Component Object Model. A component software architecture from Microsoft, which defines a structure for building program routines (objects) that can be called up and executed in a Windows environment.

Native Interface: An interface written to a specific database API (Application Programming Interface) which gives access to all of the features provided by the database vendor. It is typically more robust and faster than using ODBC, JDBC, or OLE DB (which are database translation interfaces.)

Database Gateway: A product that provides a connection to a database through a proprietary interface. Typically gateways make the database being connected to "look" like the gateway vendor's database (e.g.., Oracle's gateway to IBM DB2 makes DB2 look like an Oracle database). Database gateways translate SQL calls into a standard format known as Format and Protocol (FAP). One of the most popular gateway architectures is IBM's Distributed Relational Database Access (DRDA).

CORBA: Common Object Request Broker Architecture. A standard from the Object Management Group (OMG) for communicating between distributed objects (objects are self-contained software modules). CORBA provides a way to execute programs (objects) written in different programming languages running on different platforms no matter where they reside in the network. CORBA is suited for three-tier (or more) client/server applications, where processing occurring in one computer requires processing to be performed in another. CORBA is often described as an "object bus" or "software bus," because it is a software-based communications interface through which objects are located and accessed.

Business Intelligence Portal: A corporate portal that enables users to query and produce reports on enterprise-wide databases. The term was coined by Information Advantage, makers of the MyEureka software, which was the first to combine BI software with a corporate portal. Information Advantage was acquired by Sterling Software, which was in turn acquired by Computer Associates.

SSL: Secure Sockets Layer. A leading security protocol on the Internet. When an SSL session is started, the browser sends its public key to the server so that the server can securely send a secret key to the browser. The browser and server exchange data via secret key encryption during that session. There are also other key technologies in this area, such as PKI and SET.