# Forecasting SQL Query Cost at Twitter

arXiv: https://arxiv.org/abs/2204.05529 · HTML: https://mlsanity.com/paper/2204.05529/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 35 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 2 of 2)

- [Adaptive Huber Regression](https://mlsanity.com/paper/1706.06991/) (2017)
- [Plan-Structured Deep Neural Network Models for Query Performance Prediction](https://mlsanity.com/paper/1902.00132/) (2019)

## Built on by (most cited first, 5 of 5)

- [Can LLMs substitute SQL? Comparing Resource Utilization of Querying LLMs versus Traditional Relational Databases](https://mlsanity.com/paper/2404.08727/) (2024)
- [SafeLoad: Efficient Admission Control Framework for Identifying Memory-Overloading Queries in Cloud Data Warehouses](https://mlsanity.com/paper/2601.01888/) (2026)
- [LearnedWMP: Workload Memory Prediction Using Distribution of Query Templates](https://mlsanity.com/paper/2401.12103/) (2024)
- [On Enhancing Root Cause Analysis with SQL Summaries for Failures in Database Workload Replays at SAP HANA](https://mlsanity.com/paper/2412.13679/) (2024)
- [ScaleSense: Cost-Intelligent Scaling Framework via Learned Resource Estimation in Alibaba AnalyticDB](https://mlsanity.com/paper/2608.07945/) (2026)

## Abstract (arXiv)

With the advent of the Big Data era, it is usually computationally expensive to calculate the resource usages of a SQL query with traditional DBMS approaches. Can we estimate the cost of each query more efficiently without any computation in a SQL engine kernel? Can machine learning techniques help to estimate SQL query resource utilization? The answers are yes. We propose a SQL query cost predictor service, which employs machine learning techniques to train models from historical query request logs and rapidly forecasts the CPU and memory resource usages of online queries without any computation in a SQL engine. At Twitter, infrastructure engineers are maintaining a large-scale SQL federation system across on-premises and cloud data centers for serving ad-hoc queries. The proposed service can help to improve query scheduling by relieving the issue of imbalanced online analytical processing (OLAP) workloads in the SQL engine clusters. It can also assist in enabling preemptive scaling. Additionally, the proposed approach uses plain SQL statements for the model training and online prediction, indicating it is both hardware and software-agnostic. The method can be generalized to broader SQL systems and heterogeneous environments. The models can achieve 97.9% accuracy for CPU usage prediction and 97% accuracy for memory usage prediction.