# An End-to-End Learning-based Cost Estimator

arXiv: https://arxiv.org/abs/1906.02560 · HTML: https://mlsanity.com/paper/1906.02560/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Jun 2019.

## Built on by (most cited first, 30 of 43)

- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation](https://mlsanity.com/paper/2109.05877/) (2021)
- [Lero: A Learning-to-Rank Query Optimizer](https://mlsanity.com/paper/2302.06873/) (2023)
- [A Unified Deep Model of Learning from both Data and Queries for Cardinality Estimation](https://mlsanity.com/paper/2107.12295/) (2021)
- [Balsa: Learning a Query Optimizer Without Expert Demonstrations](https://mlsanity.com/paper/2201.01441/) (2022)
- [Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction](https://mlsanity.com/paper/2201.00561/) (2022)
- [A Survey on Advancing the DBMS Query Optimizer: Cardinality Estimation, Cost Model, and Plan Enumeration](https://mlsanity.com/paper/2101.01507/) (2021)
- [FactorJoin: A New Cardinality Estimation Framework for Join Queries](https://mlsanity.com/paper/2212.05526/) (2022)
- [A Unified Transferable Model for ML-Enhanced DBMS](https://mlsanity.com/paper/2105.02418/) (2021)
- [One Model to Rule them All: Towards Zero-Shot Learning for Databases](https://mlsanity.com/paper/2105.00642/) (2021)
- [Efficient Deep Learning Pipelines for Accurate Cost Estimations Over Large Scale Query Workload](https://mlsanity.com/paper/2103.12465/) (2021)
- [Database Workload Characterization with Query Plan Encoders](https://mlsanity.com/paper/2105.12287/) (2021)
- [RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systems](https://mlsanity.com/paper/2503.04252/) (2025)
- [AMED: Automatic Mixed-Precision Quantization for Edge Devices](https://mlsanity.com/paper/2205.15437/) (2022)
- [Duet: efficient and scalable hybriD neUral rElation undersTanding](https://mlsanity.com/paper/2307.13494/) (2023)
- [FeatNavigator: Automatic Feature Augmentation on Tabular Data](https://mlsanity.com/paper/2406.09534/) (2024)
- [Learned Graph Rewriting with Equality Saturation: A New Paradigm in Relational Query Rewrite and Beyond](https://mlsanity.com/paper/2407.12794/) (2024)
- [A Practical Theory of Generalization in Selectivity Learning](https://mlsanity.com/paper/2409.07014/) (2024)
- [OpenMLDB: A Real-Time Relational Data Feature Computation System for Online ML](https://mlsanity.com/paper/2501.08591/) (2025)
- [Can Deep Neural Networks Predict Data Correlations from Column Names?](https://mlsanity.com/paper/2107.04553/) (2021)
- [Glue: Adaptively Merging Single Table Cardinality to Estimate Join Query Size](https://mlsanity.com/paper/2112.03458/) (2021)
- [ML-Powered Index Tuning: An Overview of Recent Progress and Open Challenges](https://mlsanity.com/paper/2308.13641/) (2023)
- [A Learned Cost Model-based Cross-engine Optimizer for SQL Workloads](https://mlsanity.com/paper/2506.02802/) (2025)
- [Cost Trade-offs of Reasoning and Non-Reasoning Large Language Models in Text-to-SQL](https://mlsanity.com/paper/2512.22364/) (2025)
- [AI-Driven Research for Databases](https://mlsanity.com/paper/2604.06566/) (2026)
- [Electra: Conditional Generative Model based Predicate-Aware Query Approximation](https://mlsanity.com/paper/2201.12420/) (2022)
- [A Novel Technique for Query Plan Representation Based on Graph Neural Nets](https://mlsanity.com/paper/2405.04814/) (2024)
- [CAMAL: Optimizing LSM-trees via Active Learning](https://mlsanity.com/paper/2409.15130/) (2024)
- [CardOOD: Robust Query-driven Cardinality Estimation under Out-of-Distribution](https://mlsanity.com/paper/2412.05864/) (2024)
- [DistJoin: A Decoupled Join Cardinality Estimator based on Adaptive Neural Predicate Modulation](https://mlsanity.com/paper/2503.08994/) (2025)

## Abstract (arXiv)

Cost and cardinality estimation is vital to query optimizer, which can guide the plan selection. However traditional empirical cost and cardinality estimation techniques cannot provide high-quality estimation, because they cannot capture the correlation between multiple columns. Recently the database community shows that the learning-based cardinality estimation is better than the empirical methods. However, existing learning-based methods have several limitations. Firstly, they can only estimate the cardinality, but cannot estimate the cost. Secondly, convolutional neural network (CNN) with average pooling is hard to represent complicated structures, e.g., complex predicates, and the model is hard to be generalized. To address these challenges, we propose an effective end-to-end learning-based cost estimation framework based on a tree-structured model, which can estimate both cost and cardinality simultaneously. To the best of our knowledge, this is the first end-to-end cost estimator based on deep learning. We propose effective feature extraction and encoding techniques, which consider both queries and physical operations in feature extraction. We embed these features into our tree-structured model. We propose an effective method to encode string values, which can improve the generalization ability for predicate matching. As it is prohibitively expensive to enumerate all string values, we design a patten-based method, which selects patterns to cover string values and utilizes the patterns to embed string values. We conducted experiments on real-world datasets and experimental results showed that our method outperformed baselines.