The Elements of Event Stream Processing

Figure 5

To deliver the benefits of ESP requires a range of architectural elements that include event-processing servers, event data man agement, event visualization, system management, ESB integration, and ESP development tools. Conceptually, these basic elements are similar to their static computing counterparts. A static system might, for example, use Java as a programming language, Visual Basic for the graphical user interface, and SQL to access data, and a relational database for storage. In the world of ESP, these are more likely to be a Java-based event-programming language (EPL), a dashboard for visualisation, and an event data store for event storage and retrieval. In total, there are seven elements that deliver a comprehensive ESP software platform:

1.  Event Processing Engine - Just as an application server is the brain of a multi-tier architecture so an event-processing engine is the brain of an event-driven SOA. The event-processing engine monitors multiple event streams, identifying relationships among events by applying ESP rules that are expressed with an EPL. It may utilize non-event data (sourced from an RDBMS) or previously received events (from an event data store) as part of its analysis. Its output represents "derived" or "complex" events that can be sent to other applications, often via an ESB.

2.  Source and Sink Stream Adapters - Event-driven SOA emits events via messages on an ESB; tapping into those event streams via source (inbound) and sink (outbound) stream adapters. For deployments without an ESB, other (often application-specific) transports can be used to receive and/or distribute the events. For example, in event-processing applications involving RFID (radio frequency identification), Application Level Events (ALE) is a standard interface for capturing radio signals from RFID tags. In financial event stream applications, trading systems integrate with various exchanges via market data feeds (e.g., Reuters) or order management protocols such as FIX.

3.  Event Development Tools - An EPL lets developers express complex event processing (CEP) rules that detect temporal, causal, and spatial relationships among events. Recent innovations in ESP have resulted in EPL tools that let business users create, deploy, and modify rules themselves - further aiding "agility" by enabling business users to participate in the expression of business logic. An in-depth EPL example and its ability to discover complex relationships among events is discussed below.

4.  Event Visualization and Multi-Channel Output - ESP processing immediately notifies users through alerts on a computer screen dashboard. But dashboards are just one vehicle for event delivery. Multi-channel output enables an ESP processing system to send e-mail, control automated devices via software interfaces, issue messages via pagers and other wireless devices - or send data to kiosks for consumers to see. 5.  Event Data Stores - The characteristics of event data - both in volume and type - suggest the need of a new class of data management infrastructure to meet the demands of stream-processing applications. Such event stores should support configurations that 1) capture every event (meaningful or not) that is carried by the ESB and 2) capture filtered subsets of events as determined by ESP logic to be meaningful or 3) new "derived" events ("complex" events) that are created from other events. Rather than store events in RDBMS tables that focus on static attributes such event data stores will typically organize and retrieve data using time as the key value.

6.  Relational Database Caching - As the processing engine operates on event streams, it often needs access to structured data to inform decision-making. For example, to rebook passengers correctly, the Delta Nervous System may have to access passenger itineraries to determine if rerouting is practical. Relational caching technology provides in-memory access to cached relational data sets. By caching this data, an ESP platform can avoid repetitive SQL requests to a relational server when checking the impact of flight rerouting on a plane's entire passenger manifest.

7.  Management, Fault Tolerance, State Recovery - Enterprise-class "abilities" - reliability, manageability, and scalability - are critical elements in any software platform and ESP is no different. Although typical management and monitoring facilities are critical for an effective ESP architecture, it poses some unique management challenges. For example, since each event-processing engine holds its own view event state, distributed state management and recovery become more challenging.

Each of these elements of ESP can be explored in depth elsewhere; the balance of this article focuses on the intersection of ESP and ESB together with the foundational programming and architectural issues of ESP.

The ESB Is the Stream
ESB service endpoints can both generate and respond to asynchronous events. Though the ESB provides the integration backbone, it doesn't inherently deliver the analytics layer that can oversee and understand all the resulting events that are spawned by such an architecture. ESP adds insight to ESB-generated events.

Figure 6

In the example below, the trading desks of an investment bank are integrated with an ESB. Equities (stocks), government bonds, and foreign exchange are traded by independent, loosely coupled systems, whose service-oriented interfaces are exposed on an ESB. A key benefit of an ESB is that it normalizes the enterprise event stream into XML; JMS, application events, and FTP events are transformed into a normalized form by the ESB fabric and these events are delivered in streams that can be analysed for patterns in real-time by ESP. In this example, each trading desk executes trades and emits events via the ESB that represent trades. The ESP processing engine monitors these events, as well as other external events, and provide a real-time check that the firm's overall risk (a compilation of equities, bonds, and foreign exchange) hasn't been exceeded. Allowable trading levels gyrate according to real-time activity; when one trading desk takes an aggressive position, another might have a conservative one; combined, the risk is acceptable. However, if two desks take an aggressive position at the same time, they must be instructed to modify that position, perhaps by sending an event that automatically limits the amount of a security it can trade. If the desks don't alter their risk positions quickly enough, ESP can send an event that reports the situation to a risk management authority. ESP can monitor and moderate the real-time ebb and flow of risk in a large firm.

Figure 7

The use of ESP and an ESB together has been described as an advanced stage of maturity of an SOA in Sonic Software's "SOA Maturity Model." How do ESP and an ESB combine to make an effective event-driven architecture work?

ESP - Finding ESB Event Patterns Using Causality and Temporal Constraints
To illustrate how the ESB and ESP can work together, let's examine a specific example: credit card fraud detection. The goal is to monitor purchase activity in the system and capture authorization requests that can be analysed in real-time to detect fraudulent activity. Such operations fully express the three-stage event-processing paradigm of 1) monitor, 2) analyse, and 3) act.

First, we need electronic access to the events on the ESB. Source event streams, show below as "Merchant A" and "Merchant B," represent purchase activity events delivered via messages on the ESB to the event engine.

Second, we need rules that act on those events. Since ESP engines process events asynchronously, events can come from anywhere, be of any type, and be received in any order. An EPL uses events, their time of occurrence, any causal relationship among events, and event abstraction as its fundamental elements, rather than the structured data and relational algebra of SQL. An EPL processes queries "partially" - for example, when A and B are true then if C occurs in N seconds, take some action. An example of one rule is shown below.

Figure 8

This EPL code begins with an event filter ("ON credit_payment (user)"). This statement instructs the engine to monitor events on the ESB that represent charges to a credit card; as events flow through the ESB, this event pattern is partially satisfied. Next, the "FOLLOWED-BY" statement instructs the ESP engine to monitor subsequent credit_payment events against the same account ("user"). If three charges occur in two minutes then the code has identified a potentially fraudulent activity. Though not depicted above, an example of including "spatial" logic might be to flag any charges that fall outside a pattern of purchases geographically or flag a subsequent charge if the geographic distance between the purchase locations suggests that the legitimacy of one of the charges is improbable, if not impossible. These examples illustrate the first central concept of ESP: the inference of causality. ESP has inferred from the relationship among the charges that one or more of the charges were caused by caused by fraud.