# Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation

arXiv: https://arxiv.org/abs/2108.12211 · HTML: https://mlsanity.com/paper/2108.12211/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 9 of 9)

- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Low-Level Augmented Bayesian Optimization for Finding the Best Cloud VM](https://mlsanity.com/paper/1712.10081/) (2017)
- [Micky: A Cheaper Alternative for Selecting Cloud Instances](https://mlsanity.com/paper/1803.05587/) (2018)
- [Tuneful: An Online Significance-Aware Configuration Tuner for Big Data Analytics](https://mlsanity.com/paper/2001.08002/) (2020)
- [C3O: Collaborative Cluster Configuration Optimization for Distributed Data Processing in Public Clouds](https://mlsanity.com/paper/2107.13317/) (2021)
- [Scout: An Experienced Guide to Find the Best Cloud Configuration](https://mlsanity.com/paper/1803.01296/) (2018)
- [Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts](https://mlsanity.com/paper/2107.13921/) (2021)
- [Parallelization in Scientific Workflow Management Systems](https://mlsanity.com/paper/1303.7195/) (2013)
- [Towards Collaborative Optimization of Cluster Configurations for Distributed Dataflow Jobs](https://mlsanity.com/paper/2011.07965/) (2020)

## Built on by (most cited first, 2 of 2)

- [Magpie: Automatically Tuning Static Parameters for Distributed File Systems using Deep Reinforcement Learning](https://mlsanity.com/paper/2207.09298/) (2022)
- [Perona: Robust Infrastructure Fingerprinting for Resource-Efficient Big Data Analytics](https://mlsanity.com/paper/2211.08227/) (2022)

## Abstract (arXiv)

Distributed dataflow systems like Spark and Flink enable the use of clusters for scalable data analytics. While runtime prediction models can be used to initially select appropriate cluster resources given target runtimes, the actual runtime performance of dataflow jobs depends on several factors and varies over time. Yet, in many situations, dynamic scaling can be used to meet formulated runtime targets despite significant performance variance. This paper presents Enel, a novel dynamic scaling approach that uses message propagation on an attributed graph to model dataflow jobs and, thus, allows for deriving effective rescaling decisions. For this, Enel incorporates descriptive properties that capture the respective execution context, considers statistics from individual dataflow tasks, and propagates predictions through the job graph to eventually find an optimized new scale-out. Our evaluation of Enel with four iterative Spark jobs shows that our approach is able to identify effective rescaling actions, reacting for instance to node failures, and can be reused across different execution contexts.