# Evaluation of Load Prediction Techniques for Distributed Stream Processing

arXiv: https://arxiv.org/abs/2108.04749 · HTML: https://mlsanity.com/paper/2108.04749/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 38 references that are in the map.
- First posted on arXiv 10 Aug 2021.

## Builds on (most cited first, 6 of 6)

- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Deep Learning with Long Short-Term Memory for Time Series Prediction](https://mlsanity.com/paper/1810.10161/) (2018)
- [Online Supervised Learning for Traffic Load Prediction in Framed-ALOHA Networks](https://mlsanity.com/paper/1907.11064/) (2019)
- [A Utilization Model for Optimization of Checkpoint Intervals in Distributed Stream Processing Systems](https://mlsanity.com/paper/1911.11915/) (2019)
- [Effectively Testing System Configurations of Critical IoT Analytics Pipelines](https://mlsanity.com/paper/2102.06094/) (2021)
- [Chiron: Optimizing Fault Tolerance in QoS-aware Distributed Stream Processing Jobs](https://mlsanity.com/paper/2102.06170/) (2021)

## Built on by (most cited first, 1 of 1)

- [Towards a Proactive Autoscaling Framework for Data Stream Processing at the Edge using GRU and Transfer Learning](https://mlsanity.com/paper/2507.14597/) (2025)

## Abstract (arXiv)

Distributed Stream Processing (DSP) systems enable processing large streams of continuous data to produce results in near to real time. They are an essential part of many data-intensive applications and analytics platforms. The rate at which events arrive at DSP systems can vary considerably over time, which may be due to trends, cyclic, and seasonal patterns within the data streams. A priori knowledge of incoming workloads enables proactive approaches to resource management and optimization tasks such as dynamic scaling, live migration of resources, and the tuning of configuration parameters during run-times, thus leading to a potentially better Quality of Service. In this paper we conduct a comprehensive evaluation of different load prediction techniques for DSP jobs. We identify three use-cases and formulate requirements for making load predictions specific to DSP jobs. Automatically optimized classical and Deep Learning methods are being evaluated on nine different datasets from typical DSP domains, i.e. the IoT, Web 2.0, and cluster monitoring. We compare model performance with respect to overall accuracy and training duration. Our results show that the Deep Learning methods provide the most accurate load predictions for the majority of the evaluated datasets.