# ML-Powered Index Tuning: An Overview of Recent Progress and Open Challenges

arXiv: https://arxiv.org/abs/2308.13641 · HTML: https://mlsanity.com/paper/2308.13641/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 82 references that are in the map.
- First posted on arXiv 25 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [Neo: A Learned Query Optimizer](https://mlsanity.com/paper/1904.03711/) (2019)
- [Bao: Learning to Steer Query Optimizers](https://mlsanity.com/paper/2004.03814/) (2020)
- [An End-to-End Learning-based Cost Estimator](https://mlsanity.com/paper/1906.02560/) (2019)
- [BestConfig: Tapping the Performance Potential of Systems via Automatic Configuration Tuning](https://mlsanity.com/paper/1710.03439/) (2017)
- [Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources](https://mlsanity.com/paper/1802.10233/) (2018)
- [Plan-Structured Deep Neural Network Models for Query Performance Prediction](https://mlsanity.com/paper/1902.00132/) (2019)
- [DB-BERT: a Database Tuning Tool that "Reads the Manual"](https://mlsanity.com/paper/2112.10925/) (2021)
- [Qd-tree: Learning Data Layouts for Big Data Analytics](https://mlsanity.com/paper/2004.10898/) (2020)
- [SkinnerDB: Regret-Bounded Query Evaluation via Reinforcement Learning](https://mlsanity.com/paper/1901.05152/) (2019)
- [Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings](https://mlsanity.com/paper/2002.12393/) (2020)
- [Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction](https://mlsanity.com/paper/2201.00561/) (2022)
- [UDO: Universal Database Optimization using Reinforcement Learning](https://mlsanity.com/paper/2104.01744/) (2021)
- [The Case for Automatic Database Administration using Deep Reinforcement Learning](https://mlsanity.com/paper/1801.05643/) (2018)
- [Comprehensive and Efficient Workload Compression](https://mlsanity.com/paper/2011.05549/) (2020)
- [Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Data](https://mlsanity.com/paper/2210.05508/) (2022)
- [Uncertainty Aware Query Execution Time Prediction](https://mlsanity.com/paper/1408.6589/) (2014)
- [Database Workload Characterization with Query Plan Encoders](https://mlsanity.com/paper/2105.12287/) (2021)
- [No DBA? No regret! Multi-armed bandits for index tuning of analytical and HTAP workloads with provable guarantees](https://mlsanity.com/paper/2108.10130/) (2021)
- [DBA bandits: Self-driving index tuning under ad-hoc, analytical workloads with safety guarantees](https://mlsanity.com/paper/2010.09208/) (2020)
- [Deploying a Steered Query Optimizer in Production at Microsoft](https://mlsanity.com/paper/2210.13625/) (2022)

## Built on by (most cited first, 2 of 2)

- [A New Paradigm in Tuning Learned Indexes: A Reinforcement Learning Enhanced Approach](https://mlsanity.com/paper/2502.05001/) (2025)
- [Budget-aware Query Tuning: An AutoML Perspective](https://mlsanity.com/paper/2404.00137/) (2024)

## Abstract (arXiv)

The scale and complexity of workloads in modern cloud services have brought into sharper focus a critical challenge in automated index tuning -- the need to recommend high-quality indexes while maintaining index tuning scalability. This challenge is further compounded by the requirement for automated index implementations to introduce minimal query performance regressions in production deployments, representing a significant barrier to achieving scalability and full automation. This paper directs attention to these challenges within automated index tuning and explores ways in which machine learning (ML) techniques provide new opportunities in their mitigation. In particular, we reflect on recent efforts in developing ML techniques for workload selection, candidate index filtering, speeding up index configuration search, reducing the amount of query optimizer calls, and lowering the chances of performance regressions. We highlight the key takeaways from these efforts and underline the gaps that need to be closed for their effective functioning within the traditional index tuning framework. Additionally, we present a preliminary cross-platform design aimed at democratizing index tuning across multiple SQL-like systems -- an imperative in today's continuously expanding data system landscape. We believe our findings will help provide context and impetus to the research and development efforts in automated index tuning.