Supervision : Pierre SENS

Co-supervision : Luciana Arantes

Dynamic Adaptation in Stream Processing System

Stream processing systems (SPS) dynamically process input events over a set of operators. As the input is generally not a constant stream, work in the literature proposes to dynamically replicate SPS operators, in order to reduce processing bottlenecks induced by fluctuations.
Here, we present two approaches for dynamically modifying the number of replicas of an operator. A first reactive approach relies on several metrics. A second approach using a predictive model analyzes input rate variation, operator execution time and queued events to dynamically define the number of replicas required for each operator. We also propose a Storm-based implementation of these two approaches, which does not require reconfiguration on restart when the number of operator replicas changes.

2021-2024 Publications