Streams and CEP

Stream Processing frameworks like Apache Storm, Apache Spark Streaming, etc. are widely used to process realtime data. These distributed stream processors are used to tackle the Velocity dimension of Big Data. Continuous dataflows are generally directed-acyclic graphs, composed of tasks processing messages with some user-logic, and sending output messages to other tasks. Many of the mission-critical applications like smart power grid, smart transportation, etc. ingest and process messages in realtime. Update to such long-running applications may be needed, for example, bug fixes, feature enhancements, etc. Currently, the application suffers a significant downtime while performing such dataflow updates. This downtime is unacceptable for mission-critical applications. We are investigating new update strategies as well as enhancing existing continuous dataflow update strategies.


Sensors that generate immense volumes of data are growing both in physical and virtual space. They generate continuous streaming data that constitutes to the aspect of Big Data. Complex Event Processing (CEP) facilitates real time event patterns to extract meaningful information from data streams. CEP engines accept continuous queries and detect patterns on event streams. Analysis of streaming data is needed to make meaningful decisions. Processing and visualization of real time streaming data is highly important.  Visualizing the streaming Big Data helps to communicate information clearly and efficiently in the form of graphs and charts. This pictorial representation helps the end users to understand the situation better. This technique communicates data in the form of visual objects.The visual representation of the insights gained from analysis should get updated as the data changes and display the newer results. We are investigating on the visual analytics of streaming Big Data in real-time.

Relevant Publications

  • T. Sharma and Y. Simmhan, “Online-Update Strategies for Distributed Stream Processing in Apache Storm,” at 7th Student Research Symposium, in IEEE Conference on High Performance Computing (HiPC), 2014
  • N. Govindarajan, Y. Simmhan, N. Jamadagni and P. Misra, “Event Processing across Edge and Cloud for Internet of Things Applications,” at 20th International Conference on Management of Data (COMAD), 2014