Processing Complex Events (During These, oh well, Complex Times) - Part III

Part I of this blog series introduced the concept of complex event processing (CEP) and possible needs for CEP software applications. One such broad CEP platform, Progress Apama, has been offered by Progress Software Coporation after acquiring formerly independent Apama LTD in 2005.

Part II then discussed Apama’s current state of affairs and its real-life deployments at companies outside its traditional stronghold of capital markets and algorithmic trading. So, what makes this product particularly attractive?

Progress Apama Differentiators

I believe that Progress Apama wins in competitive situations for several reasons, starting with being a complete platform for building CEP-based applications.  For example, in contrast to many other offerings, users can develop the application within Apama, rather than depend upon external programming languages like C++ and Java for creating business logic.

While most CEP vendors have focused to a large degree on the data manipulation services associated with processing streaming data (broadly categorized as filtering and aggregation), Progress Apama also incorporates its integrated development environment (IDE) and event-processing language (EPL), MonitorScript. Apama offers Java class libraries as well for its own EPL. The core of the Apama CEP platform is the CEP engine called CEP Correlator, which is a dedicated event “processor” where companies place their business logic.

To create business rules, the MonitorScript language can handle declarative event expressions, imperative logic expressions, and temporal operators. Apama’s proprietary EPL supports determinism and has control over determining events. The language can also cross-correlate many heterogeneous events. In addition, the CEP engine can accommodate programs written in Java or C++ (and handled via a respective virtual machine [VM] or plug-ins) if users prefer.

The aforementioned Apama IDE helps with enforcing best practices, performance, and accuracy, while also accommodating vertically focused components or “accelerators” (e.g., a broad capital markets framework). The environment provides simultaneous support for different tiers of users (with different levels of sophistication and varying needs).

To that end, business users (e.g., traders) will likely work in dashboards that were developed within Apama's Dashboard Studio, which offers an array of graphical widgets and rapid connections to event streams. Power users (e.g., business analysts) will likely spend time in Apama's Event Modeler in order to create composite event services via interactive controls. Finally, IT users will use Apama's Developer Studio, which features an Eclipse-based user interface (UI) and rich EPL as well as the aforementioned Dashboard Studio.

As mentioned in Part II, Apama also offers graphical dashboards for monitoring CEP applications, and these dashboards are tightly integrated with the underlying execution engine. The Apama Dashboard is a ready-built graphic-rich front end that customers often build themselves out of Java. The dashboard can execute in local (intranet) and Internet configurations.

Dashboard Studio lets customers define their own rules for handling data, and configure how the information will be displayed, whether it is in a pie chart, bar chart, XY chart, or some other display. This caters to rich information visualization tools such as charts, trends, and drilldowns, either with visual abstraction or with details where and if required. The UI can be deployed as a “thick” desktop client, Web-based client, or portal.

In addition to offering development tools for both business users and IT staff (with both a graphical modeling tool, i.e., Event Modeler, and an Eclipse-based developer environment), Apama offers the ability for the IT developer to create components, called SmartBlocks. These blocks encapsulate lower-level language operations to make them available for the business user within Event Modeler. Developers can also build SmartBlocks from within Event Modeler if necessary.

This user-based diversification and the “different strokes for different folks” approach (interactive builders for speed/power users vs. custom builders for developers) inspires users’ confidence and helps with instituting corporate-wide security restrictions (a safety net). In a nutshell, Progress Software offers tools for all constituencies and the bridge between them.

Moreover, Apama offers a sophisticated integration framework for connecting the CEP execution engine with event streams of many different kinds, with a range of over 40 packaged adapters to integrate with external systems. To offer only some standard connections examples, let me mention Reuters Trading Exchange (RTEx), TIBCO Software’s TIB/ETX middleware, and Financial Information Exchange (FIX).

Apama's integration framework also supports Java Message Service (JMS) and is open to several relational database management systems (RDBMS). The integration framework consists of the transport interface (to multiple event sources), the codec layer (to normalize diverse message formats), the Extensible Markup Language (XML) mapping layer (to transform messages into the Apama format), and the CEP Correlator interface. The layered framework allows users to quickly build new connectivity to in-house networks and new technologies, using different integration paths (e.g., via enterprise service bus [ESB] or point to point).

Furthermore, Apama offers a testing environment that allows companies to analyze existing or prospective applications against streaming data that has been captured in a time-series database. By offering a comprehensive platform, Progress Software makes it easier for users to create, test, visualize, execute, deploy, integrate, and evolve applications.

A single model for Research Studio and Development Studio promotes support for rapid evolution and prototyping of the application. In this unified environment users can more effectively build an application in the Apama logic that can then analyze and act on that streaming data for testing and compliance purposes.

Apama Event Store and Research Studio offer event-based tools to backtest (play back) historic data, while providing temporal consistency and optimization scenarios. Last but not least, though Progress Apama is not exclusive to capital markets, the company’s experience in that vertical area means that it has the domain expertise, field knowledge, and commitment to customers that often surpasses other offerings.

What Will the Future Bring?

Certainly, in some of the most demanding arenas, especially within capital markets, Apama leads the way in CEP with rapid application development (RAD) tools for both business and IT users, CEP best practices, a comprehensive CEP product suite, and CEP solution accelerators. Still, the vendor acknowledges that the CEP concept and use is only at the “tipping point” and that much work remains when it comes to taking its capital market experience into new markets.

In the foreseeable future, Progress Apama will focus on delivering enhanced CEP-ESB integration, new solution accelerator releases (e.g. a Futures Spreader solution), automated multicore processor scalability, and an enhanced CEP debugger. Progress Apama's blogger Louis Lovas' recent blog entry offers a peek preview of what is ahead for Apama for those who follow CEP and Progress Apama.

Given Progress Software’s apparent forays in the realm of on-demand and software as a service (SaaS), another logical question is whether any such moves are planned for Apama. In fact, the product currently supports a number of hosted deployments. In some instances, Apama is used by a sell-side bank to provide trading algorithms to its buy-side clients.  That is not an uncommon implementation model for trading systems. As an example, Progress can point to customers like SEB (the Scandinavian bank) and Agora Corretora, the largest brokerage in Brazil.

Other customers use Apama in a similar fashion, though they have not publicly announced that they use Apama in such a manner. Apama has also established relationships with the NYSE Euronext and Bit Systems (Borsa Italiana Technology) exchanges in which Apama is hosted in those environments, allowing those exchanges to make available algorithmic trading to clients as a way to attract trader order flow.

Also, Apama has established a partnership with CQG, a popular charting tool for futures trading. In this arrangement, CQG hosts Apama in its environment and CQG customers can employ Apama to automatically execute trades via integration with the CQG application.

Caveat: CEP Not a Cure-all

But given the worldwide heightened anxiety (and even anger) due to the blazing global financial crisis, we should not expect Apama to perform miracles of total financial risk management. Progress Software acknowledges that Apama (and the entire CEP industry if you will) is only at an early stage in using sophisticated algorithms to map business processes to help increase the visibility into risks of financial instruments that are dispersed across institutions or globally. Since so many organizations in the current economic crisis are not yet aware of who owns what pieces of which risk, the problems with this kind of risk-sharing are of a more qualitative than quantitative nature.

The power of CEP is in its capacity to perform quantitative market monitoring and to do it with a light touch that doesn’t have to be obtrusive to the normal course of business. While algorithms and rules sets for seeing patterns in data are not that difficult to grasp (for users that intimately know their business), it is still up to knowledgeable users to encapsulate that know-how and harness the system to look for anomalies and useful information. Hence, today’s CEP tools are not quite capable of handling the kind of risk management that Wall Street needs.

To be fair, there are elements of risk (e.g., real-time market risk, operational risk) for which Apama is quite suitable and which Wall Street does need to manage.  What Apama is not tuned to do is the sophisticated financial risk models that a product like Algorithmics does.

After all, financial institutions are not individuals, and cannot be easily subjected to something analogous to retail credit practices, where payment information is compiled and updated constantly by credit reporting agencies and where missing a payment on a mortgage, car, doctor bill, or a credit card can impact overall credit history and force re-pricing of asset loans. The problem is much more complex than that, and time alone will tell when someone will be able to encapsulate it as a hefty set of CEP rules and algorithms.

CEP systems are based on rules and are not able to self-learn and discover correlations in an undirected manner (i.e., like knowledge management systems) or conduct the sophisticated data mining that predictive analytics can do. CEP is neither based on artificial intelligence (AI) methods such as artificial neural networks, nor on Bayesian inference, nor on other advanced algorithms.

Notable Competition in a Still-budding Market

As the early 2008 article from IT-Director estimated, in addition to Progress Apama, StreamBase Systems and Aleri and Coral8 (soon to be one firm) are other notable players in the capital markets segment. Moreover, in some environments that need strong data warehouse back ends for CEP systems (possibly to handle huge radio frequency identification [RFID] data streams), there is Sybase with its Real-Time Analytics Platform, Vertica, and Kx Systems.

Given the ongoing global war on terror (even if the Obama Administration might name it differently), another logical market for CEP is intelligence and surveillance services, though this is still not that exploited outside the US. Here, StreamBase and AgentLogic are the leading vendors. But even more than in the secretive financial services business, the nature of the applications’ use is highly secret in this "James Bond" environment, and no vendor can easily provide references and brag about its experience. Still, the opportunity is there for all vendors to penetrate non-US intelligence markets.

Certainly there have been a number of implementations in transportation and logistics (where TIBCO Software has been doing well) but my gut feeling is that the next major market might be in process manufacturing and in other adjacent asset-intensive sectors (such as power stations and utilities). Demonstrable benefits can be achieved in productivity gains by employing CEP in these environments (Progress Apama, as mentioned in Part II, and Avaya iSphere can both quote examples) and, perhaps most important, shop-floor environments tend to understand the concept of events in order to reduce lead times and cycles. IBM AptSoft and Oracle CEP are other players that should not be neglected either across the board.

Even though there are so many examples of possible use for CEP, there are not that many established markets yet. A total market size of around few hundred customers and few hundred million dollars to-date might mean that CEP is still only nibbling at the possible opportunity pie today, but the possibilities are enormous. Of course, all vendors, including Progress Apama, need to pull out all the stops to provide new solution accelerators (i.e., to capture the industry know-how into rules), engage new industry partners, and spread the message what CEP can (and cannot) do.

Therefore, dear readers, what are your views, comments, opinions, etc. about Apama’s applicability, and about the CEP software markets in general? We would also be interested in your experiences with this software category (if you are an existing user), and your general interest to evaluate these solutions as prospective customers.
comments powered by Disqus