ML-Powered Index Tuning: An Overview of Recent Progress and Open Challenges
Not on the map. Why?
Abstract (arXiv)
The scale and complexity of workloads in modern cloud services have brought into sharper focus a critical challenge in automated index tuning -- the need to recommend high-quality indexes while maintaining index tuning scalability. This challenge is further compounded by the requirement for automated index implementations to introduce minimal query performance regressions in production deployments, representing a significant barrier to achieving scalability and full automation. This paper directs attention to these challenges within automated index tuning and explores ways in which machine learning (ML) techniques provide new opportunities in their mitigation. In particular, we reflect on recent efforts in developing ML techniques for workload selection, candidate index filtering, speeding up index configuration search, reducing the amount of query optimizer calls, and lowering the chances of performance regressions. We highlight the key takeaways from these efforts and underline the gaps that need to be closed for their effective functioning within the traditional index tuning framework. Additionally, we present a preliminary cross-platform design aimed at democratizing index tuning across multiple SQL-like systems -- an imperative in today's continuously expanding data system landscape. We believe our findings will help provide context and impetus to the research and development efforts in automated index tuning.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 20 of its 82 references that are in the map. First posted on arXiv 25 Aug 2023.
Builds on
20 of 82 references.
References · 20
- 2019Neo: A Learned Query Optimizercited by 80
- 2020Bao: Learning to Steer Query Optimizerscited by 62
- 2019An End-to-End Learning-based Cost Estimatorcited by 43
- 2017BestConfig: Tapping the Performance Potential of Systems via Automatic Configuration Tuningcited by 25
- 2018Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sourcescited by 25
- 2019Plan-Structured Deep Neural Network Models for Query Performance Predictioncited by 24
- 2021DB-BERT: a Database Tuning Tool that "Reads the Manual"cited by 19
- 2020Qd-tree: Learning Data Layouts for Big Data Analyticscited by 18
- 2019SkinnerDB: Regret-Bounded Query Evaluation via Reinforcement Learningcited by 17
- 2020Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findingscited by 16
- 2022Zero-Shot Cost Models for Out-of-the-box Learned Cost Predictioncited by 16
- 2021UDO: Universal Database Optimization using Reinforcement Learningcited by 11
8 more
- 2018The Case for Automatic Database Administration using Deep Reinforcement Learningcited by 8
- 2020Comprehensive and Efficient Workload Compressioncited by 6
- 2022Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Datacited by 6
- 2014Uncertainty Aware Query Execution Time Predictioncited by 4
- 2021Database Workload Characterization with Query Plan Encoderscited by 4
- 2021No DBA? No regret! Multi-armed bandits for index tuning of analytical and HTAP workloads with provable guaranteescited by 4
- 2020DBA bandits: Self-driving index tuning under ad-hoc, analytical workloads with safety guaranteescited by 3
- 2022Deploying a Steered Query Optimizer in Production at Microsoftcited by 2
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.