# LOCAT: Low-Overhead Online Configuration Auto-Tuning of Spark SQL Applications

arXiv: https://arxiv.org/abs/2203.14889 · HTML: https://mlsanity.com/paper/2203.14889/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Mar 2022.

## Built on by (most cited first, 6 of 6)

- [VDTuner: Automated Performance Tuning for Vector Data Management Systems](https://mlsanity.com/paper/2404.10413/) (2024)
- [Rover: An online Spark SQL tuning service via generalized transfer learning](https://mlsanity.com/paper/2302.04046/) (2023)
- [AI-Driven Research for Databases](https://mlsanity.com/paper/2604.06566/) (2026)
- [Towards General and Efficient Online Tuning for Spark](https://mlsanity.com/paper/2309.01901/) (2023)
- [HPC Application Parameter Autotuning on Edge Devices: A Bandit Learning Approach](https://mlsanity.com/paper/2501.01057/) (2025)
- [Centrum: Model-based Database Auto-tuning with Minimal Distributional Assumptions](https://mlsanity.com/paper/2510.22734/) (2025)

## Abstract (arXiv)

Spark SQL has been widely deployed in industry but it is challenging to tune its performance. Recent studies try to employ machine learning (ML) to solve this problem, but suffer from two drawbacks. First, it takes a long time (high overhead) to collect training samples. Second, the optimal configuration for one input data size of the same application might not be optimal for others. To address these issues, we propose a novel Bayesian Optimization (BO) based approach named LOCAT to automatically tune the configurations of Spark SQL applications online. LOCAT innovates three techniques. The first technique, named QCSA, eliminates the configuration-insensitive queries by Query Configuration Sensitivity Analysis (QCSA) when collecting training samples. The second technique, dubbed DAGP, is a Datasize-Aware Gaussian Process (DAGP) which models the performance of an application as a distribution of functions of configuration parameters as well as input data size. The third technique, called IICP, Identifies Important Configuration Parameters (IICP) with respect to performance and only tunes the important ones. As such, LOCAT can tune the configurations of a Spark SQL application with low overhead and adapt to different input data sizes. We employ Spark SQL applications from benchmark suites TPC-DS, TPC-H, and HiBench running on two significantly different clusters, a four-node ARM cluster and an eight-node x86 cluster, to evaluate LOCAT. The experimental results on the ARM cluster show that LOCAT accelerates the optimization procedures of the state-of-the-art approaches by at least 4.1x and up to 9.7x; moreover, LOCAT improves the application performance by at least 1.9x and up to 2.4x. On the x86 cluster, LOCAT shows similar results to those on the ARM cluster.