# Efficient Deep Learning Pipelines for Accurate Cost Estimations Over Large Scale Query Workload

arXiv: https://arxiv.org/abs/2103.12465 · HTML: https://mlsanity.com/paper/2103.12465/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [One weird trick for parallelizing convolutional neural networks](https://mlsanity.com/paper/1404.5997/) (2014)
- [Measuring the Effects of Data Parallelism on Neural Network Training](https://mlsanity.com/paper/1811.03600/) (2018)
- [Neo: A Learned Query Optimizer](https://mlsanity.com/paper/1904.03711/) (2019)
- [Learned Cardinalities: Estimating Correlated Joins with Deep Learning](https://mlsanity.com/paper/1809.00677/) (2018)
- [An End-to-End Learning-based Cost Estimator](https://mlsanity.com/paper/1906.02560/) (2019)
- [Efficient Segmentation: Learning Downsampling Near Semantic Boundaries](https://mlsanity.com/paper/1907.07156/) (2019)
- [Learning State Representations for Query Optimization with Deep Reinforcement Learning](https://mlsanity.com/paper/1803.08604/) (2018)
- [QuickSel: Quick Selectivity Learning with Mixture Models](https://mlsanity.com/paper/1812.10568/) (2018)
- [Towards a Hands-Free Query Optimizer through Deep Learning](https://mlsanity.com/paper/1809.10212/) (2018)
- [An Empirical Analysis of Deep Learning for Cardinality Estimation](https://mlsanity.com/paper/1905.06425/) (2019)
- [Cross-Language Learning for Program Classification using Bilateral Tree-Based Convolutional Neural Networks](https://mlsanity.com/paper/1710.06159/) (2017)
- [Facilitating SQL Query Composition and Analysis](https://mlsanity.com/paper/2002.09091/) (2020)

## Built on by (most cited first, 4 of 4)

- [Lero: A Learning-to-Rank Query Optimizer](https://mlsanity.com/paper/2302.06873/) (2023)
- [Zero-Shot Cost Models for Out-of-the-box Learned Cost Prediction](https://mlsanity.com/paper/2201.00561/) (2022)
- [Detect, Distill and Update: Learned DB Systems Facing Out of Distribution Data](https://mlsanity.com/paper/2210.05508/) (2022)
- [MICRO: A Lightweight Middleware for Optimizing Cross-store Cross-model Graph-Relation Joins [Technical Report]](https://mlsanity.com/paper/2603.13835/) (2026)

## Abstract (arXiv)

The use of deep learning models for forecasting the resource consumption patterns of SQL queries have recently been a popular area of study. With many companies using cloud platforms to power their data lakes for large scale analytic demands, these models form a critical part of the pipeline in managing cloud resource provisioning. While these models have demonstrated promising accuracy, training them over large scale industry workloads are expensive. Space inefficiencies of encoding techniques over large numbers of queries and excessive padding used to enforce shape consistency across diverse query plans implies 1) longer model training time and 2) the need for expensive, scaled up infrastructure to support batched training. In turn, we developed Prestroid, a tree convolution based data science pipeline that accurately predicts resource consumption patterns of query traces, but at a much lower cost. We evaluated our pipeline over 19K Presto OLAP queries from Grab, on a data lake of more than 20PB of data. Experimental results imply that our pipeline outperforms benchmarks on predictive accuracy, contributing to more precise resource prediction for large-scale workloads, yet also reduces per-batch memory footprint by 13.5x and per-epoch training time by 3.45x. We demonstrate direct cost savings of up to 13.2x for large batched model training over Microsoft Azure VMs.