# Towards General and Efficient Online Tuning for Spark

arXiv: https://arxiv.org/abs/2309.01901 · HTML: https://mlsanity.com/paper/2309.01901/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 88 references that are in the map.
- First posted on arXiv 5 Sep 2023.

## Builds on (most cited first, 20 of 20)

- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Scalable Global Optimization via Local Bayesian Optimization](https://mlsanity.com/paper/1910.01739/) (2019)
- [MLlib: Machine Learning in Apache Spark](https://mlsanity.com/paper/1505.06807/) (2015)
- [Constrained Efficient Global Optimization of Expensive Black-box Functions](https://mlsanity.com/paper/2211.00162/) (2022)
- [OpenBox: A Generalized Black-box Optimization Service](https://mlsanity.com/paper/2106.00421/) (2021)
- [Transfer Learning for Bayesian Optimization: A Survey](https://mlsanity.com/paper/2302.05927/) (2023)
- [BestConfig: Tapping the Performance Potential of Systems via Automatic Configuration Tuning](https://mlsanity.com/paper/1710.03439/) (2017)
- [VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition](https://mlsanity.com/paper/2107.08861/) (2021)
- [Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation](https://mlsanity.com/paper/2110.12654/) (2021)
- [Towards Dynamic and Safe Configuration Tuning for Cloud Databases](https://mlsanity.com/paper/2203.14473/) (2022)
- [Transfer Learning based Search Space Design for Hyperparameter Tuning](https://mlsanity.com/paper/2206.02511/) (2022)
- [Black or White? How to Develop an AutoTuner for Memory-based Analytics [Extended Version]](https://mlsanity.com/paper/2002.11780/) (2020)
- [OpenBox: A Python Toolkit for Generalized Black-box Optimization](https://mlsanity.com/paper/2304.13339/) (2023)
- [TransBO: Hyperparameter Optimization via Two-Phase Transfer Learning](https://mlsanity.com/paper/2206.02663/) (2022)
- [LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications](https://mlsanity.com/paper/2203.14889/) (2022)
- [A Unified and Efficient Coordinating Framework for Autonomous DBMS Tuning](https://mlsanity.com/paper/2303.05710/) (2023)
- [KEA: Tuning an Exabyte-Scale Data Infrastructure](https://mlsanity.com/paper/2106.11445/) (2021)
- [Rover: An online Spark SQL tuning service via generalized transfer learning](https://mlsanity.com/paper/2302.04046/) (2023)
- [Spark Parameter Tuning via Trial-and-Error](https://mlsanity.com/paper/1607.07348/) (2016)
- [Learning-based Automatic Parameter Tuning for Big Data Analytics Frameworks](https://mlsanity.com/paper/1808.06008/) (2018)

## Built on by (most cited first, 1 of 1)

- [SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines](https://mlsanity.com/paper/2408.04323/) (2024)

## Abstract (arXiv)

The distributed data analytic system -- Spark is a common choice for processing massive volumes of heterogeneous data, while it is challenging to tune its parameters to achieve high performance. Recent studies try to employ auto-tuning techniques to solve this problem but suffer from three issues: limited functionality, high overhead, and inefficient search. In this paper, we present a general and efficient Spark tuning framework that can deal with the three issues simultaneously. First, we introduce a generalized tuning formulation, which can support multiple tuning goals and constraints conveniently, and a Bayesian optimization (BO) based solution to solve this generalized optimization problem. Second, to avoid high overhead from additional offline evaluations in existing methods, we propose to tune parameters along with the actual periodic executions of each job (i.e., online evaluations). To ensure safety during online job executions, we design a safe configuration acquisition method that models the safe region. Finally, three innovative techniques are leveraged to further accelerate the search process: adaptive sub-space generation, approximate gradient descent, and meta-learning method. We have implemented this framework as an independent cloud service, and applied it to the data platform in Tencent. The empirical results on both public benchmarks and large-scale production tasks demonstrate its superiority in terms of practicality, generality, and efficiency. Notably, this service saves an average of 57.00% memory cost and 34.93% CPU cost on 25K in-production tasks within 20 iterations, respectively.