Towards an Adaptive Framework for Real-Time Visualization of Streaming Big Data

Abstract

Big data poses new challenges and the need for flexible, interactive, and dynamic visualization techniques. Existing approaches, especially in enterprise data visualization with static graphics or interactive dashboards, are limited at the scale of big data, given the volume and diversity of data to consider. Streaming data further compounds on the problem with the need for real-time analytics and visualizations. On the data acquisition and collection side of things, traditional business analytics platforms are being extended with support for technologies such as Apache Spark for improvement in performance. However, for real-time data visualization for streaming data, it is necessary to go beyond Apache Spark with in-memory processing and new data visualization idioms. We propose a framework for the dynamic visualization of real-time streaming big data, resilient to both its volume and rate of change. Some of the different directions we explore include: (a) the efficient processing and consumption of streaming data; (b) the automated detection of relevant changes in the data stream, highlighting entities that merit a detailed analysis; (c) the choice of the best idioms to visualize big data, possibly leading to the development of new visualization idioms; (d) real-time visualization changes.