# LearnedWMP: Workload Memory Prediction Using Distribution of Query Templates

arXiv: https://arxiv.org/abs/2401.12103 · HTML: https://mlsanity.com/paper/2401.12103/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 82 references that are in the map.
- First posted on arXiv 22 Jan 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Recent Advances in Recurrent Neural Networks](https://mlsanity.com/paper/1801.01078/) (2017)
- [Learned Cardinalities: Estimating Correlated Joins with Deep Learning](https://mlsanity.com/paper/1809.00677/) (2018)
- [Mean Absolute Percentage Error for regression models](https://mlsanity.com/paper/1605.02541/) (2016)
- [Learning Theory for Distribution Regression](https://mlsanity.com/paper/1411.2066/) (2014)
- [An End-to-End Learning-based Cost Estimator](https://mlsanity.com/paper/1906.02560/) (2019)
- [Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation](https://mlsanity.com/paper/2109.05877/) (2021)
- [Plan-Structured Deep Neural Network Models for Query Performance Prediction](https://mlsanity.com/paper/1902.00132/) (2019)
- [Distribution-Free Distribution Regression](https://mlsanity.com/paper/1302.0082/) (2013)
- [Deep Distribution Regression](https://mlsanity.com/paper/1903.06023/) (2019)
- [Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings](https://mlsanity.com/paper/2002.12393/) (2020)
- [Bayesian Approaches to Distribution Regression](https://mlsanity.com/paper/1705.04293/) (2017)
- [The Case for Automatic Database Administration using Deep Reinforcement Learning](https://mlsanity.com/paper/1801.05643/) (2018)
- [Forecasting SQL Query Cost at Twitter](https://mlsanity.com/paper/2204.05529/) (2022)
- [Uncertainty Aware Query Execution Time Prediction](https://mlsanity.com/paper/1408.6589/) (2014)
- [Database Workload Characterization with Query Plan Encoders](https://mlsanity.com/paper/2105.12287/) (2021)
- [Real-time Workload Pattern Analysis for Large-scale Cloud Databases](https://mlsanity.com/paper/2307.02626/) (2023)
- [Sibyl: Forecasting Time-Evolving Query Workloads](https://mlsanity.com/paper/2401.03723/) (2024)
- [Coefficient-based Regularized Distribution Regression](https://mlsanity.com/paper/2208.12427/) (2022)

## Abstract (arXiv)

In a modern DBMS, working memory is frequently the limiting factor when processing in-memory analytic query operations such as joins, sorting, and aggregation. Existing resource estimation approaches for a DBMS estimate the resource consumption of a query by computing an estimate of each individual database operator in the query execution plan. Such an approach is slow and error-prone as it relies upon simplifying assumptions, such as uniformity and independence of the underlying data. Additionally, the existing approach focuses on individual queries separately and does not factor in other queries in the workload that may be executed concurrently. In this research, we are interested in query performance optimization under concurrent execution of a batch of queries (a workload). Specifically, we focus on predicting the memory demand for a workload rather than providing separate estimates for each query within it. We introduce the problem of workload memory prediction and formalize it as a distribution regression problem. We propose Learned Workload Memory Prediction (LearnedWMP) to improve and simplify estimating the working memory demands of workloads. Through a comprehensive experimental evaluation, we show that LearnedWMP reduces the memory estimation error of the state-of-the-practice method by up to 47.6%. Compared to an alternative single-query model, during training and inferencing, the LearnedWMP model and its variants were 3x to 10x faster. Moreover, LearnedWMP-based models were at least 50% smaller in most cases. Overall, the results demonstrate the advantages of the LearnedWMP approach and its potential for a broader impact on query performance optimization.