# Towards a Proactive Autoscaling Framework for Data Stream Processing at the Edge using GRU and Transfer Learning

arXiv: https://arxiv.org/abs/2507.14597 · HTML: https://mlsanity.com/paper/2507.14597/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 4 of its 55 references that are in the map.
- First posted on arXiv 19 Jul 2025.

## Builds on (most cited first, 4 of 4)

- [Maximum Mean Discrepancy for Generalization in the Presence of Distribution and Missingness Shift](https://mlsanity.com/paper/2111.10344/) (2021)
- [Multi-Source Transfer Learning for Non-Stationary Environments](https://mlsanity.com/paper/1901.02052/) (2019)
- [Evaluation of Load Prediction Techniques for Distributed Stream Processing](https://mlsanity.com/paper/2108.04749/) (2021)
- [Learning from the Past: Adaptive Parallelism Tuning for Stream Processing Systems](https://mlsanity.com/paper/2504.12074/) (2025)

## Abstract (arXiv)

Processing data at high speeds is becoming increasingly critical as digital economies generate enormous data. The current paradigms for timely data processing are edge computing and data stream processing (DSP). Edge computing places resources closer to where data is generated, while stream processing analyzes the unbounded high-speed data in motion. However, edge stream processing faces rapid workload fluctuations, complicating resource provisioning. Inadequate resource allocation leads to bottlenecks, whereas excess allocation results in wastage. Existing reactive methods, such as threshold-based policies and queuing theory scale only after performance degrades, potentially violating SLAs. Although reinforcement learning (RL) offers a proactive approach through agents that learn optimal runtime adaptation policies, it requires extensive simulation. Furthermore, predictive machine learning models face online distribution and concept drift that minimize their accuracy. We propose a three-step solution to the proactive edge stream processing autoscaling problem. Firstly, a GRU neural network forecasts the upstream load using real-world and synthetic DSP datasets. Secondly, a transfer learning framework integrates the predictive model into an online stream processing system using the DTW algorithm and joint distribution adaptation to handle the disparities between offline and online domains. Finally, a horizontal autoscaling module dynamically adjusts the degree of operator parallelism, based on predicted load while considering edge resource constraints. The lightweight GRU model for load predictions recorded up to 1.3% SMAPE value on a real-world data set. It outperformed CNN, ARIMA, and Prophet on the SMAPE and RMSE evaluation metrics, with lower training time than the computationally intensive RL models.