# PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems

arXiv: https://arxiv.org/abs/1810.06115 · HTML: https://mlsanity.com/paper/1810.06115/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Oct 2018.

## Built on by (most cited first, 22 of 22)

- [Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs](https://mlsanity.com/paper/2204.12013/) (2022)
- [GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edge](https://mlsanity.com/paper/2201.07705/) (2022)
- [The Synergy of Speculative Decoding and Batching in Serving Large Language Models](https://mlsanity.com/paper/2310.18813/) (2023)
- [End-to-end Optimization of Machine Learning Prediction Queries](https://mlsanity.com/paper/2206.00136/) (2022)
- [FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guarantees](https://mlsanity.com/paper/2402.18789/) (2024)
- [Accelerating Deep Learning Inference via Learned Caches](https://mlsanity.com/paper/2101.07344/) (2021)
- [Pisces: Efficient Federated Learning via Guided Asynchronous Training](https://mlsanity.com/paper/2206.09264/) (2022)
- [Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Vision](https://mlsanity.com/paper/2205.11913/) (2022)
- [FENXI: Deep-learning Traffic Analytics at the Edge](https://mlsanity.com/paper/2105.11738/) (2021)
- [Multi-model Machine Learning Inference Serving with GPU Spatial Partitioning](https://mlsanity.com/paper/2109.01611/) (2021)
- [CascadeServe: Unlocking Model Cascades for Inference Serving](https://mlsanity.com/paper/2406.14424/) (2024)
- [Machine Learning Systems are Bloated and Vulnerable](https://mlsanity.com/paper/2212.09437/) (2022)
- [Symphony: Optimized DNN Model Serving using Deferred Batch Scheduling](https://mlsanity.com/paper/2308.07470/) (2023)
- [IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency](https://mlsanity.com/paper/2308.12871/) (2023)
- [Serving Deep Learning Model in Relational Databases](https://mlsanity.com/paper/2310.04696/) (2023)
- [Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines](https://mlsanity.com/paper/2405.11191/) (2024)
- [Cocktail: Leveraging Ensemble Learning for Optimized Model Serving in Public Cloud](https://mlsanity.com/paper/2106.05345/) (2021)
- [Automatic and Efficient Customization of Neural Networks for ML Applications](https://mlsanity.com/paper/2310.04685/) (2023)
- [Keep Your Friends Close: Leveraging Affinity Groups to Accelerate AI Inference Workflows](https://mlsanity.com/paper/2312.11488/) (2023)
- [UnifiedNN: Efficient Neural Network Training on the Cloud](https://mlsanity.com/paper/2408.01331/) (2024)
- [HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Models](https://mlsanity.com/paper/2504.17449/) (2025)
- [DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMs](https://mlsanity.com/paper/2607.28848/) (2026)

## Abstract (arXiv)

Machine Learning models are often composed of pipelines of transformations. While this design allows to efficiently execute single model components at training time, prediction serving has different requirements such as low latency, high throughput and graceful performance degradation under heavy load. Current prediction serving systems consider models as black boxes, whereby prediction-time-specific optimizations are ignored in favor of ease of deployment. In this paper, we present PRETZEL, a prediction serving system introducing a novel white box architecture enabling both end-to-end and multi-model optimizations. Using production-like model pipelines, our experiments show that PRETZEL is able to introduce performance improvements over different dimensions; compared to state-of-the-art approaches PRETZEL is on average able to reduce 99th percentile latency by 5.5x while reducing memory footprint by 25x, and increasing throughput by 4.7x.