# Optimal Resource Allocation for Serverless Queries

arXiv: https://arxiv.org/abs/2107.08594 · HTML: https://mlsanity.com/paper/2107.08594/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 27 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 4 of 4)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [SimGNN: A Neural Network Approach to Fast Graph Similarity Computation](https://mlsanity.com/paper/1808.05689/) (2018)
- [Cost Models for Big Data Query Processing: Learning, Retrofitting, and Our Findings](https://mlsanity.com/paper/2002.12393/) (2020)
- [A Comparative Exploration of ML Techniques for Tuning Query Degree of Parallelism](https://mlsanity.com/paper/2005.08439/) (2020)

## Built on by (most cited first, 2 of 2)

- [Deploying a Steered Query Optimizer in Production at Microsoft](https://mlsanity.com/paper/2210.13625/) (2022)
- [Predictive Price-Performance Optimization for Serverless Query Processing](https://mlsanity.com/paper/2112.08572/) (2021)

## Abstract (arXiv)

Optimizing resource allocation for analytical workloads is vital for reducing costs of cloud-data services. At the same time, it is incredibly hard for users to allocate resources per query in serverless processing systems, and they frequently misallocate by orders of magnitude. Unfortunately, prior work focused on predicting peak allocation while ignoring aggressive trade-offs between resource allocation and run-time. Additionally, these methods fail to predict allocation for queries that have not been observed in the past. In this paper, we tackle both these problems. We introduce a system for optimal resource allocation that can predict performance with aggressive trade-offs, for both new and past observed queries. We introduce the notion of a performance characteristic curve (PCC) as a parameterized representation that can compactly capture the relationship between resources and performance. To tackle training data sparsity, we introduce a novel data augmentation technique to efficiently synthesize the entire PCC using a single run of the query. Lastly, we demonstrate the advantages of a constrained loss function coupled with GNNs, over traditional ML methods, for capturing the domain specific behavior through an extensive experimental evaluation over SCOPE big data workloads at Microsoft.