Make smart and accurate
software selection decisions
Podcasts, Webinars, and Videos
Interactive Case Studies
ERGO Decision Support System
Private Label Partnerships
TEC Case Studies
Software Evaluation Reports
Meet TEC's Experts
News and Press Releases
Working at TEC
Partner with TEC
Processing Complex Events (During These, oh well, Complex Times) - Pa...
Processing Complex Events (During These, oh well, Complex Times) - Part III
April 28 2009
Part I of this blog series
introduced the concept of
complex event processing (CEP
) and possible needs for CEP software applications. One such broad CEP platform,
, has been offered by
Progress Software Coporation
after acquiring formerly independent
Part II then discussed Apama’s current state of affairs
and its real-life deployments at companies outside its traditional stronghold of
. So, what makes this product particularly attractive?
Progress Apama Differentiators
I believe that Progress Apama wins in competitive situations for several reasons, starting with being a complete platform for building CEP-based applications. For example, in contrast to many other offerings, users can develop the application within Apama, rather than depend upon external programming languages like
for creating business logic.
While most CEP vendors have focused to a large degree on the data manipulation services associated with processing streaming data (broadly categorized as filtering and aggregation), Progress Apama also incorporates its
integrated development environment (IDE)
and event-processing language (EPL),
. Apama offers Java class libraries as well for its own EPL. The core of the Apama CEP platform is the CEP engine called CEP Correlator, which is a dedicated event “processor” where companies place their business logic.
To create business rules, the MonitorScript language can handle
logic expressions, and temporal operators. Apama’s proprietary EPL supports
and has control over determining events. The language can also cross-correlate many heterogeneous events. In addition, the CEP engine can accommodate programs written in Java or C++ (and handled via a respective
virtual machine [VM]
) if users prefer.
The aforementioned Apama IDE helps with enforcing best practices, performance, and accuracy, while also accommodating vertically focused components or “accelerators” (e.g., a broad capital markets framework). The environment provides simultaneous support for different tiers of users (with different levels of sophistication and varying needs).
To that end,
(e.g., traders) will likely work in dashboards that were developed within
Apama's Dashboard Studio
, which offers an array of graphical widgets and rapid connections to event streams.
(e.g., business analysts) will likely spend time in
Apama's Event Modeler
in order to create composite event services via interactive controls. Finally, IT users will use
Apama's Developer Studio
, which features an
user interface (UI
) and rich EPL as well as the aforementioned Dashboard Studio.
As mentioned in Part II
, Apama also offers graphical dashboards for monitoring CEP applications, and these dashboards are tightly integrated with the underlying execution engine. The Apama Dashboard is a ready-built graphic-rich front end that customers often build themselves out of Java. The dashboard can execute in local (intranet) and Internet configurations.
Dashboard Studio lets customers define their own rules for handling data, and configure how the information will be displayed, whether it is in a
, or some other display. This caters to rich information visualization tools such as charts, trends, and
, either with visual abstraction or with details where and if required. The UI can be deployed as a
“thick” desktop client
, Web-based client, or
In addition to offering development tools for both business users and IT staff (with both a graphical modeling tool, i.e., Event Modeler, and an Eclipse-based developer environment), Apama offers the ability for the IT developer to create components, called SmartBlocks. These blocks encapsulate lower-level language operations to make them available for the business user within Event Modeler. Developers can also build SmartBlocks from within Event Modeler if necessary.
This user-based diversification and the “different strokes for different folks” approach (interactive builders for speed/power users vs. custom builders for developers) inspires users’ confidence and helps with instituting corporate-wide security restrictions (a safety net). In a nutshell, Progress Software offers tools for all constituencies and the bridge between them.
Moreover, Apama offers a sophisticated integration framework for connecting the CEP execution engine with event streams of many different kinds, with a range of over 40 packaged adapters to integrate with external systems. To offer only some standard connections examples, let me mention
Reuters Trading Exchange (RTEx
TIBCO Software’s TIB/ETX
Financial Information Exchange (FIX
Apama's integration framework also supports
Java Message Service (JMS
) and is open to several
relational database management systems (RDBMS
). The integration framework consists of the transport interface (to multiple event sources), the
diverse message formats), the
Extensible Markup Language (XML
) mapping layer (to transform messages into the Apama format), and the CEP Correlator interface. The layered framework allows users to quickly build new connectivity to in-house networks and new technologies, using different integration paths (e.g., via
enterprise service bus [ESB
point to point
Furthermore, Apama offers a
environment that allows companies to analyze existing or prospective applications against streaming data that has been captured in a time-series database. By offering a comprehensive platform, Progress Software makes it easier for users to create, test, visualize, execute, deploy, integrate, and evolve applications.
A single model for Research Studio and Development Studio promotes support for rapid evolution and prototyping of the application. In this unified environment users can more effectively build an application in the Apama logic that can then analyze and act on that streaming data for testing and compliance purposes.
Apama Event Store and Research Studio offer event-based tools to backtest (play back) historic data, while providing temporal consistency and optimization scenarios. Last but not least, though Progress Apama is not exclusive to capital markets, the company’s experience in that vertical area means that it has the domain expertise, field knowledge, and commitment to customers that often surpasses other offerings.
What Will the Future Bring?
Certainly, in some of the most demanding arenas, especially within capital markets, Apama leads the way in CEP with
rapid application development (RAD)
tools for both business and IT users, CEP best practices, a comprehensive CEP product suite, and CEP solution accelerators. Still, the vendor acknowledges that the CEP concept and use is only at the
and that much work remains when it comes to taking its capital market experience into new markets.
In the foreseeable future, Progress Apama will focus on delivering enhanced CEP-ESB integration, new solution accelerator releases (e.g. a
processor scalability, and an enhanced CEP
. Progress Apama's
blogger Louis Lovas' recent blog entry offers a peek preview of what is ahead for Apama for those who follow CEP and Progress Apama
Progress Software’s apparent forays in the realm of on-demand and software as a service (SaaS)
, another logical question is whether any such moves are planned for Apama. In fact, the product currently supports a number of hosted deployments. In some instances, Apama is used by a
bank to provide trading algorithms to its
clients. That is not an uncommon implementation model for trading systems. As an example, Progress can point to customers like
(the Scandinavian bank
, the largest brokerage in Brazil
Other customers use Apama in a similar fashion, though they have not publicly announced that they use Apama in such a manner. Apama has also established relationships with the
Bit Systems (Borsa Italiana Technology
) exchanges in which Apama is hosted in those environments, allowing those exchanges to make available algorithmic trading to clients as a way to attract trader order flow.
Also, Apama has established a partnership with
, a popular charting tool for futures trading. In this arrangement, CQG hosts Apama in its environment and CQG customers can employ Apama to automatically execute trades via integration with the CQG application.
Caveat: CEP Not a Cure-all
But given the worldwide heightened anxiety (and even anger) due to the blazing global financial crisis, we should not expect Apama to perform miracles of total financial
. Progress Software acknowledges that Apama (and the entire CEP industry if you will) is only at an early stage in using sophisticated algorithms to map business processes to help increase the
visibility into risks of financial instruments that are dispersed across institutions or globally. Since so many organizations in the current economic crisis are not yet aware of who owns what pieces of which risk
, the problems with this kind of risk-sharing are of a more qualitative than quantitative nature.
The power of CEP is in its capacity to perform quantitative market monitoring and to do it with a light touch that doesn’t have to be obtrusive to the normal course of business. While algorithms and rules sets for seeing patterns in data are not that difficult to grasp (for users that intimately know their business), it is still up to knowledgeable users to encapsulate that know-how and harness the system to look for anomalies and useful information. Hence, today’s CEP tools are not quite capable of handling the kind of risk management that Wall Street needs.
To be fair, there are elements of risk (e.g., real-time market risk, operational risk) for which Apama is quite suitable and which Wall Street does need to manage. What Apama is not tuned to do is the sophisticated financial risk models that a product like
After all, financial institutions are not individuals, and cannot be easily subjected to something analogous to retail credit practices, where payment information is compiled and updated constantly by credit reporting agencies and where missing a payment on a mortgage, car, doctor bill, or a credit card can impact overall credit history and force re-pricing of asset loans. The problem is much more complex than that, and time alone will tell when someone will be able to encapsulate it as a hefty set of CEP rules and algorithms.
CEP systems are based on rules and are not able to self-learn and discover correlations in an undirected manner (i.e., like
knowledge management systems
) or conduct the sophisticated data mining that
can do. CEP is neither based on artificial intelligence (AI) methods such as artificial
, nor on
, nor on other advanced algorithms.
Notable Competition in a Still-budding Market
As the early 2008 article from
, in addition to Progress Apama,
soon to be one firm
) are other notable players in the capital markets segment. Moreover, in some environments that need strong data warehouse back ends for CEP systems (possibly to handle huge
radio frequency identification [RFID
] data streams), there is
with its Real-Time Analytics Platform
Given the ongoing global war on terror (even if the Obama Administration might name it differently), another logical market for CEP is intelligence and surveillance services, though this is still not that exploited outside the US. Here,
are the leading vendors. But even more than in the secretive financial services business, the nature of the applications’ use is highly secret in this "James Bond" environment, and no vendor can easily provide references and brag about its experience. Still, the opportunity is there for all vendors to penetrate non-US intelligence markets.
Certainly there have been a number of implementations in transportation and logistics (where TIBCO Software has been doing well) but my gut feeling is that the next major market might be in
and in other adjacent asset-intensive sectors (such as power stations and utilities). Demonstrable benefits can be achieved in productivity gains by employing CEP in these environments (Progress Apama, as mentioned
in Part II
can both quote examples) and, perhaps most important, shop-floor environments tend to understand the concept of events in order to reduce lead times and cycles.
are other players that should not be neglected either across the board
Even though there are so many examples of possible use for CEP, there are not that many established markets yet. A total market size of around few hundred customers and few hundred million dollars to-date might mean that CEP is still only nibbling at the possible opportunity pie today, but the possibilities are enormous. Of course, all vendors, including Progress Apama, need to pull out all the stops to provide new solution accelerators (i.e., to capture the industry know-how into rules), engage new industry partners, and spread the message what CEP can (and cannot) do.
Therefore, dear readers, what are your views, comments, opinions, etc. about Apama’s applicability, and about the CEP software markets in general? We would also be interested in your experiences with this software category (if you are an existing user), and your general interest to evaluate these solutions as prospective customers.
comments powered by Disqus.
comments powered by
Interested in a better way to make software decisions?
Give us a call now: 1-800-496-1303 ext:404
Software Requirements Sets and Comparison Reports
Click here to leverage the experience of our 360 industry perspective