Complex Event Processing

A picture is worth a thousand words, correct? So, how much does a video translate to - exponentially greater number of words....

With all the attention being given to Big Data, the realization is sinking in that we are creating snapshots - which is a great step forward, but that we really need streaming video for much better pattern recognition.

I interviewed Dr. John Bates, founder of Apama and ex Cambridge professor for my next book and he talked about the "video" - the technologies which are classified as Complex Event Processing

“CEP enables real-time analytics to be performed on streaming data to show, for example, impending trends; it also allows patterns to be correlated across events in diverse data streams that can represent occurrences the business needs to know about and/or proactively respond to.

Inputs to a CEP system can include sensor data (e.g. SCADA sensors on a oil pipeline), data feeds (stock, news etc.), application data (e.g. CRM, ERP, credit card transactions etc.), location data (GPS, RFID etc.) and any other kind of feed.

CEP can shut down fraud while it is happening, push a marketing offer to a customer while they are in the right place at the right time or detect and place an algorithmic trade in microseconds, before a competitor spots the opportunity.”

The financial services industry has been an early adopter of CEP for its trading algorithms. That has moved to detecting insider trading patterns – correlating business events like earnings announcements to stock trades as an example. Now they are are adapting it on the consumer side of financial services for fraud detection.

John had tons of CEP examples in other industries - many of which will show in the book.

It's complex stuff - as the acronym suggests (though John remarkably presents his thoughts in lucid business constructs) . Deciding which signals to focus on and the correlation across them is not a skillset you find in abundance. But as in so many areas, the Polymath who can master multiple streams usually does better the Monomath.