# A Learned Cost Model-based Cross-engine Optimizer for SQL Workloads

arXiv: https://arxiv.org/abs/2506.02802 · HTML: https://mlsanity.com/paper/2506.02802/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 36 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 7 of 7)

- [An End-to-End Learning-based Cost Estimator](https://mlsanity.com/paper/1906.02560/) (2019)
- [Lero: A Learning-to-Rank Query Optimizer](https://mlsanity.com/paper/2302.06873/) (2023)
- [Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources](https://mlsanity.com/paper/1802.10233/) (2018)
- [Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction](https://mlsanity.com/paper/2201.00561/) (2022)
- [Learned Graph Rewriting with Equality Saturation: A New Paradigm in Relational Query Rewrite and Beyond](https://mlsanity.com/paper/2407.12794/) (2024)
- [Skeena: Efficient and Consistent Cross-Engine Transactions](https://mlsanity.com/paper/2108.00632/) (2021)
- [AWESOME: Empowering Scalable Data Science on Social Media Data with an Optimized Tri-Store Data System](https://mlsanity.com/paper/2112.00833/) (2021)

## Built on by (most cited first, 2 of 2)

- [Rethinking Query Optimization for Multi-Agent Systems [Vision]](https://mlsanity.com/paper/2512.11001/) (2025)
- [Bootstrapping Learned Cost Models with Synthetic SQL Queries](https://mlsanity.com/paper/2508.19807/) (2025)

## Abstract (arXiv)

Lakehouse systems enable the same data to be queried with multiple execution engines. However, selecting the engine best suited to run a SQL query still requires a priori knowledge of the query computational requirements and an engine capability, a complex and manual task that only becomes more difficult with the emergence of new engines and workloads. In this paper, we address this limitation by proposing a cross-engine optimizer that can automate engine selection for diverse SQL queries through a learned cost model. Optimized with hints, a query plan is used for query cost prediction and routing. Cost prediction is formulated as a multi-task learning problem, and multiple predictor heads, corresponding to different engines and provisionings, are used in the model architecture. This eliminates the need to train engine-specific models and allows the flexible addition of new engines at a minimal fine-tuning cost. Results on various databases and engines show that using a query optimized logical plan for cost estimation decreases the average Q-error by even 12.6% over using unoptimized plans as input. Moreover, the proposed cross-engine optimizer reduces the total workload runtime by up to 25.2% in a zero-shot setting and 30.4% in a few-shot setting when compared to random routing.