mlsanityData to 7 Oct 2026

PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems

arXiv 1810.06115 ↗First posted 14 Oct 2018cs.LG cs.DC stat.MLcited by 22

Not on the map. Why?

Abstract (arXiv)

Machine Learning models are often composed of pipelines of transformations. While this design allows to efficiently execute single model components at training time, prediction serving has different requirements such as low latency, high throughput and graceful performance degradation under heavy load. Current prediction serving systems consider models as black boxes, whereby prediction-time-specific optimizations are ignored in favor of ease of deployment. In this paper, we present PRETZEL, a prediction serving system introducing a novel white box architecture enabling both end-to-end and multi-model optimizations. Using production-like model pipelines, our experiments show that PRETZEL is able to introduce performance improvements over different dimensions; compared to state-of-the-art approaches PRETZEL is on average able to reduce 99th percentile latency by 5.5x while reducing memory footprint by 25x, and increasing throughput by 4.7x.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 6; 2024: 4; 2025: 1; 2026: 1.

22 papers.

Sort:

By area:

  1. 2022Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNscited by 49
  2. 2022GEMEL: Model Merging for Memory-Efficient, Real-Time Video Analytics at the Edgecited by 32
  3. 2023The Synergy of Speculative Decoding and Batching in Serving Large Language Modelscited by 19
  4. 2022End-to-end Optimization of Machine Learning Prediction Queriescited by 16
  5. 2024FlexLLM: Token-Level Co-Serving of LLM Inference and Finetuning with SLO Guaranteescited by 9
  6. 2021Accelerating Deep Learning Inference via Learned Cachescited by 8
  7. 2022Pisces: Efficient Federated Learning via Guided Asynchronous Trainingcited by 8
  8. 2022Deep Learning Workload Scheduling in GPU Datacenters: Taxonomy, Challenges and Visioncited by 6
  9. 2021FENXI: Deep-learning Traffic Analytics at the Edgecited by 4
  10. 2021Multi-model Machine Learning Inference Serving with GPU Spatial Partitioningcited by 4
  11. 2024CascadeServe: Unlocking Model Cascades for Inference Servingcited by 4
  12. 2022Machine Learning Systems are Bloated and Vulnerablecited by 2
10 more
  1. 2023Symphony: Optimized DNN Model Serving using Deferred Batch Schedulingcited by 2
  2. 2023IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiencycited by 2
  3. 2023Serving Deep Learning Model in Relational Databasescited by 2
  4. 2024Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelinescited by 2
  5. 2021Cocktail: Leveraging Ensemble Learning for Optimized Model Serving in Public Cloudcited by 1
  6. 2023Automatic and Efficient Customization of Neural Networks for ML Applicationscited by 0
  7. 2023Keep Your Friends Close: Leveraging Affinity Groups to Accelerate AI Inference Workflowscited by 0
  8. 2024UnifiedNN: Efficient Neural Network Training on the Cloudcited by 0
  9. 2025HMI: Hierarchical Knowledge Management for Efficient Multi-Tenant Inference in Pretrained Language Modelscited by 0
  10. 2026DeltaServe: Host-Agnostic Co-Serving of Inference and Fine-Tuning for LLMscited by 0