^Title

^Abstract

By ^Author

^Sidebars

Application integration is a complex problem. Thus it's appropriate to take the long view of application integration technology—how it exists now and, perhaps more important, the direction it is likely to take in the next few years.

Although there have been some notable successes in addressing application integration, these successes have been narrow and limited. The reality is, most application projects exist at the entry level. We have yet to see the real-time coupling of thousands of applications, which is the goal for many trading communities and intra-enterprise application integration problem domains. This reality should not necessarily be discouraging. As with any complex problem, once it is broken down to its components, the solution becomes simply the aggregation of a number of solution sets. In this case, it's a combination of various approaches and several types of technology.

The world of application integration is no different from the larger world of technology—it is advancing and changing rapidly. Ironically, as the technology changes, so does the problem it is designed to solve—morphing from the very simple to the very complex, even as it moves from a departmental problem to an enterprise-wide problem and, ultimately, to a trading community problem. Consequently, few companies have been able to get ahead of the "application integration curve." Short of a complete solution, they have yet to discover the full promise of application integration.

As the problem grows, so does the potential benefit of the solution. The technology continues to respond to the perceived need. In this context, the pursuit of application integration is like chasing the tail of a growing beast. For now, that beast—and a great deal of work—remains ahead of us. But rest assured, a solution will be found and the once unimaginable benefits of application integration will become an everyday reality.

Problem Domains Change

As suggested earlier, no sooner is a "traditional" application integration problem solved (such as application-to-application and database-to-database integration), than organizations apply the developed application integration expertise and technology to more complex, but more rewarding, business issues.

In fact, it is not unreasonable to see in the development of the solution the growth of the problem. This is both natural and intelligent. As systems are integrated within the enterprise, it makes good business sense to take that experience and technology and apply it to other opportunities that influence the bottom line.

B2B Applications Emerging. Supply chain integration is an old issue, and one well served by new application integration techniques. By using familiar approaches and technologies, organizations can bring together very different systems that exist in different companies and allow these systems to easily share information.

Using automation to integrate the supply chain is nothing new. However, the application of new technology and standards, such as message brokers and XML, to the problem provides many additional opportunities to address this challenge effectively and efficiently.

Moving from EDI to XML. Although electronic data interchange (EDI) represents a sound solution, its complexity and expense largely doom it—even though it will remain a point of integration for some time. Because EDI does not generally support real-time integration between applications, since it is traditionally batch-oriented, XML provides a much more efficient approach and, as such, a much less expensive approach.

Many organizations have opted for "none of the above." That is, they've chosen not to use either XML or EDI. Instead, they have adopted some other message standard (e.g., Java Message Service, or JMS) or a proprietary communications mechanism. The hodgepodge of technologies and standards that has resulted from these various in-house decisions will remain in place as organizations seek to address the supply chain integration problem.

Moving From Data-Oriented To Application-Oriented Integration. Another clear trend is the movement away from data-oriented integration to application-oriented (application interface-oriented and method-oriented) integration. Data-oriented integration provides an inexpensive mechanism to integrate applications, because in most instances there is no need to change the applications.

Although data-oriented integration (replicating data between two or more databases) provides a functional solution for many application integration problem domains, the integration of both application services and application methods generally provides more value in the long run. The downside, at least with method-oriented integration (the binding together of applications at both the program logic and data levels) is that this approach makes it necessary to change the source and target applications, or worse, in a number of instances, to create a new application (a composite application). Data-oriented integration is a much more difficult sell within trading communities, but works well within enterprise application integration (EAI) problem domains. Still, the upside of data-oriented integration is that it is consistent with the "baby-step" tactic most enterprises are comfortable with when implementing integration solutions.

In general, application integration solutions tend to be created in a series of small, low-risk steps. This type of implementation can be successful from department to enterprise to trading community, but never the other way around—from trading community to department.

Data-oriented application integration provides most organizations with a low-risk option for getting application integration under control. After that, as more time and money become available and the appetite for risk increases, enterprises can plot a strategy to "step up" to application-oriented integration.

Performance and Scalability

As enterprises move forward with application integration, they are finding that many solutions simply won't scale. Thus, the focus recently has been on designing solutions that scale to support the integration of hundreds of applications, versus less than six.

Performance and scalability issues must be part of the application integration design from the ground up, ensuring that a set of integrated systems performs well. Many organizations fail to adequately address performance and scalability until it is too late.

When considering performance engineering and application integration, organizations must take into account complex issues such as message rates, transactions per second, and interface performance. The most reliable approach to performance in the design of an integration solution is to select the technology and then create a simulation model to make an educated assessment of how well it will perform. Using simulation tools, an organization can determine the time it will take for a message or a transaction to move between applications. This simulation must test the various entities of the system at the component (e.g., server) and system levels (e.g., integrated systems) to ensure the overall performance of the application integration solution. Just as a chain is no stronger than its weakest link, an integration solution is no more efficient than its slowest-performing component. Testing identifies problem components before the solution is implemented.

Scalability differs from performance in important ways. The scalability of the system refers to the application integration solution's ability to support an increasing number of source and target systems and transaction rates. Unfortunately, scalability is rarely considered until very late in the game—often, too late. Like performance, scalability requires an architect's input from the ground up.

Making an application integration solution scale requires selecting the correct approaches and enabling technology for the problem domain. For example, if the message rate is going to be a limiting factor, select a technology that can distribute message processing among any number of connected message brokers. If transaction load and database access integrity are the limiting factors, then transactional middleware may be a better fit. No hard-and-fast rule exists for designing a system to scale. The first step is to determine the requirements, and then "back" the appropriate architecture and technology into those requirements. Putting the technology first is a very costly example of putting the cart ahead of the horse. Again, simulation tools and performance models are the first stop in determining the requirements of scalability issues.

Now and in the foreseeable future, one-stop shopping is simply not an application integration reality.

Middleware Vendor Approaches

While there are various approaches organizations can take to application integration, vendors themselves have some interesting views on the subject—which leads to the inevitable question, Who is right? As with everything else in the application integration domain, the answer is not black and white.

Application integration is a combination of problems. Each organization and trading community has its own set of integration issues to address. Therefore, it is next to impossible to find a single technological solution set that can be applied universally. Each application integration solution generally requires products from several different vendors. Now and in the foreseeable future, one-stop shopping is simply not an application integration reality.

Although vendor approaches to application integration vary considerably, there are some general categories, including:

• data-oriented
• application integration-oriented
• process integration-oriented
• transaction-oriented
• distributed object-oriented

Data-Oriented Integration

Vendors that promote the data-oriented approach to application integration argue that integration should occur between the databases—that is, databases should be viewed as the primary points of integration. However, even in data-oriented application integration, many approaches exist. It is no surprise that each vendor is quick to promote its particular solution. Examples of vendors offering solutions in this category include: Data Junction Corp., Austin, Texas; and DataMirror Corp., Markham, Ontario, with its Constellar acquisition.

Data-oriented solutions can be grouped into two categories: data replication and data federation.

Data Replication. Data replication is simply moving data between two or more databases. These databases can come from the same vendor or from many vendors. They can even be databases that employ different models. The fundamental requirement of database replication is that it accounts for the differences between database models and database schemas by providing the infrastructure to exchange data. Solutions that provide such infrastructures are plentiful and inexpensive. Most relational database vendors, including Sybase and Oracle, provide database replication services in their product offerings. These replication engines typically exist within the database engines, at either end.

Many database-oriented middleware solutions on the market provide database replication services as well. Replication services are accomplished by placing a layer of software between two or more databases. On one side, the data is extracted from the source database or databases, and on the other side, the data is placed in the target database or databases. Many of these solutions also provide transformation services—the ability to adjust the schemas and the content so they make sense to the target database.

The advantages of database replication are simplicity and low cost. Database replication is easy to implement, and the technology is cheap to purchase and install. Unfortunately, these advantages are quickly lost if methods need to be bound to the data or if methods are shared along with the data. If these requirements exist, method-sharing solutions such as transaction-oriented or application integration-oriented application integration must be considered.

Data Federation. Database federation is the integration of multiple databases and database models into a single unified view of the databases. To put it another way, database federations are virtual enterprise databases that consist of many real physical databases. Although database federation has been around for some time, the solution set has been perfected only recently.

Database federation software places a layer of software (middleware) between the physical distributed databases and the applications that view the data. This layer connects to the back-end databases by using available interfaces and maps the physical databases to a virtual database model that exists only in the software. The application uses this virtual database to access the required information. The database federation handles the collection and distribution of the data, as needed, to the physical databases. The advantage of using this software is its ability to bind many different data types into a unified model that supports information exchange.

Database federation allows access to any connected database in the enterprise through a single, well-defined interface. This is the most elegant solution to the data-oriented application integration problem. Unlike replication, this solution does not require changes to the source or target applications. Still, changes do have to be made at the application level to support federated database software, because different interfaces are being used to access a different database model (the virtual database).

Application Integration-Oriented

Application integration-oriented product solutions use well-defined application interfaces to focus on the integration of both packaged and custom applications. This approach supports the data, method, application interface, and user interface application integration types. Interest in integrating popular ERP applications (e.g., SAP, PeopleSoft, and Baan) has made this the most exciting application integration sector. (Although distributed object-oriented and transaction-oriented solutions may be applied to this space, message broker vendors are promoting their products as the preferred solution.)

Message brokers support application integration-oriented solutions by providing adapters to connect to as many custom or packaged applications as possible. They also connect to technology solutions that include middleware and screen scrapers as points of integration.

The efficient integration of many different types of applications defines the primary advantage of using application integration-oriented products. In just days, it's possible to connect a SAP R/3 application to a Baan application; the application integration-oriented solution accounts for differences among schema, content, and application semantics by translating the information moving between the systems on the fly. Moreover, this type of solution can be used as a database replication solution, able to connect to application interfaces.

The downside (there is always a downside) to using application interface-oriented products is that there is little regard for business logic and methods within the source or target systems—logic and methods that may be relevant to a particular integration effort. In such a case, transaction or distributed object-oriented solutions (composite applications or a pure method-oriented approach) probably make the better choice. Ultimately, application interface-oriented technology will learn to share methods and information, perhaps by joining forces with transaction-oriented or distributed object-oriented solutions. However, for now, organizations have to make an either/or decision.

Vendors offering products with this approach to integration include: CrossWorlds Software, Burlingame, Calif.; IBM, Armonk, N.Y.; New Era of Networks Inc., Englewood, Colo; SAGA Software, Reston, Va.; SeeBeyond (formerly STC), Monrovia, Calif.; and TIBCO Software Inc., Palo Alto, Calif.

Process Integration-Oriented

Process integration-oriented products layer a set of easily defined and centrally managed processes on top of existing processes within a set of enterprise applications. The goal is to bring together relevant processes found in a trading community or in an internal organization to obtain the maximum amount of value while supporting the flow of information and logic between these processes. These products view the middleware, or the plumbing, as a commodity, and provide easy-to-use visual interfaces for binding these processes together.

In reality, process integration is another layer of value resting upon existing application integration solutions, which include message brokers, application servers, distributed objects, and other middleware layers. Process integration offers a mechanism to bind disparate processes together and to create process-to-process solutions that automate tasks once performed manually. However, by diminishing the importance of the plumbing, vendors can easily lose sight of the larger picture. In reality, no single application integration vendor has solved the plumbing issues. Ultimately, the solution to these issues will be delivered by a combination of process integration and middleware vendors. Thus, the binding of middleware and process integration tools represents the future of application integration.

Vendors offering process integration products include: Vitria Technology Inc., Sunnyvale, Calif.; BEA Systems Inc., San Jose, Calif., with the acquisition of NCR's Top End solution; and Versata Inc., Oakland, Calif., with its acquisition of Verve Inc.

Transaction-Oriented

Transaction-oriented middleware products use the notion of a transaction—with connectors to back-end systems—to integrate applications. This is a clear method-oriented application integration approach. Examples include TP monitors and application servers. Although transaction-oriented products can be used for data integration, method-oriented integration is where these products shine. These products enable organizations to create common methods (transactions, really) and share those methods among many connected applications.

Transaction-oriented products offer the advantages of scalability and reliability. However, these products are intrusive to applications exchanging information within a trading community—requiring many changes to source and target systems—and success demands a significant amount of coding. As we go forward, we will see transaction-oriented middleware combining forces with messaging and message brokers, providing application integration architects and developers with the best of method-oriented, data-oriented, and application interface-oriented application integration.

Vendors with solutions taking this approach include BEA and IBM.

Distributed Object-Oriented

Like transaction-oriented products that exist in the method-oriented type of integration, distributed objects (such as those based on COM+ or CORBA standards) also provide application integration architects and developers with an opportunity to share methods. The elegance of the distributed object architecture, the built-in communications, and the ability to support the distributed model have led to the suggestion that this is the ultimate application integration technology. But nothing is precisely as it seems in the world of application integration projects. Distributed objects still have a long way to go before they can support most large-scale integration projects. They continue to fall short in supporting transactionality, messaging, and easy-to-use tools for building and deploying distributed object application integration solutions.

Vendors in this space include Iona Technologies, Waltham, Mass., and Inprise Corp., Scotts Valley, Calif.

Technologies Join Forces

These technologies represent different approaches to application integration. Most application integration problem domains use most types of application integration (method, application interface, data, and process integration) when integrating an enterprise. Therefore, organizations must select the appropriate technology that works for each type. Wishing for a single technology and a single vendor for all types is just that—wishing.

Developers and architects may use any number of tools to pass information between databases, supporting data-oriented application integration. These tools include newer message brokers as well as more traditional data replication software. In the application interface type, message brokers do a better job connecting to and moving information into and out of packaged applications. That's the upside. The downside is that all message brokers fall short in implementing method-oriented application integration or creating composite applications.

Message brokers are not designed to house, or share, application logic. Instead, they provide rudimentary rules engines to support operations, such as the identification, transformation, and routing of messages to the appropriate systems. If this is the application integration requirement, then application servers are the right choice, providing the ability to integrate many different systems by sharing common business logic and thus information.

At this point, it should be clear that message brokers and application servers are complementary and converging. Although message brokers do a more than adequate job in providing event-driven, asynchronous access to many different types of systems, application servers do a much better job in providing the infrastructure to share common logic. Both technologies integrate the enterprise—solve the application integration problem—but do so in very different ways (e.g., application servers are good at supporting portal-oriented integration, while message brokers are not). Although both application servers and message brokers use the middle-tier integration approach, application servers are front-end and application development focused, while message brokers are back-end, operations, and process-oriented.

Today most message broker vendors are partnering with application server vendors to ensure that, at least in the short term, they'll have a solution for the method type. The end result of these partnerships is sure to be the creation of hybrid products that support both application server and message-brokering features. The speed and approach will vary greatly from vendor to vendor, with larger message broker vendors purchasing smaller, more vulnerable application server vendors, and larger application server companies consuming the smaller message broker players. At the same time, some application server vendors will incorporate message-brokering features into their products, and some message broker vendors will learn to provide products more like application servers.

The integration of application servers and message brokers will result in a hybrid "superproduct" that may be called an "application and integration server." This product will offer a location for shared application logic and the ability to build composite applications as well as the ability to access any system using a synchronous or asynchronous communications model. Moreover, this new product will be able to account for the difference in application semantics, schema, and content.

Application Integration:
Clearly the Future

Enter application integration—along with an opportunity to finally integrate all of these disparate systems with minimal impact on the applications and the way an enterprise does business. Application integration provides a clear competitive advantage for most industries, an advantage that includes the ability to do business at light speed, along with the ability to satisfy customer demand in record time (by using automated processes instead of paper, faxes, and humans). We are truly moving forward into a digital economy, where business runs within and between computers, where everything is automated, and customers learn to expect no less than instantaneous access to information.

The future is clear. Unfortunately, the approaches and technology behind application integration are still in their infancy. The ultimate goal of application integration is to bind all trading community systems together in such a way that any application can access any method or any piece of data without delay to support any business process. For now, however, the knowledge necessary to fully integrate trading communities is still lacking.

David S. Linthicum is an internationally known EAI and e-business expert. His latest book is B2B Application Integration: e-Business-Enable Your Enterprise (Addison Wesley Longman). E-mail him at linthicum@worldnet.att.net.