# Desiderata for next generation of ML model serving

arXiv: https://arxiv.org/abs/2210.14665 · HTML: https://mlsanity.com/paper/2210.14665/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 46 references that are in the map.
- First posted on arXiv 26 Oct 2022.

## Builds on (most cited first, 15 of 15)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Stealing Machine Learning Models via Prediction APIs](https://mlsanity.com/paper/1609.02943/) (2016)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [Sustainable AI: Environmental Implications, Challenges and Opportunities](https://mlsanity.com/paper/2111.00364/) (2021)
- [Challenges in Deploying Machine Learning: a Survey of Case Studies](https://mlsanity.com/paper/2011.09926/) (2020)
- [I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defences](https://mlsanity.com/paper/2206.08451/) (2022)
- [Algorithms that Remember: Model Inversion Attacks and Data Protection Law](https://mlsanity.com/paper/1807.04644/) (2018)
- [Monitoring and explainability of models in production](https://mlsanity.com/paper/2007.06299/) (2020)
- [Context-Aware Drift Detection](https://mlsanity.com/paper/2203.08644/) (2022)
- [DeeprETA: An ETA Post-processing System at Scale](https://mlsanity.com/paper/2206.02127/) (2022)
- [An Empirical Evaluation of Flow Based Programming in the Machine Learning Deployment Context](https://mlsanity.com/paper/2204.12781/) (2022)
- [Beyond Batch Processing: Towards Real-Time and Streaming Big Data](https://mlsanity.com/paper/1403.3375/) (2014)
- [Assessing Software Privacy using the Privacy Flow-Graph](https://mlsanity.com/paper/2209.02948/) (2022)

## Built on by (most cited first, 5 of 5)

- [Dataflow graphs as complete causal graphs](https://mlsanity.com/paper/2303.09552/) (2023)
- [Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems](https://mlsanity.com/paper/2304.10892/) (2023)
- [Machine Learning Systems: A Survey from a Data-Oriented Perspective](https://mlsanity.com/paper/2302.04810/) (2023)
- [IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency](https://mlsanity.com/paper/2308.12871/) (2023)
- [Causal fault localisation in dataflow systems](https://mlsanity.com/paper/2304.11987/) (2023)

## Abstract (arXiv)

Inference is a significant part of ML software infrastructure. Despite the variety of inference frameworks available, the field as a whole can be considered in its early days. This position paper puts forth a range of important qualities that next generation of inference platforms should be aiming for. We present our rationale for the importance of each quality, and discuss ways to achieve it in practice. We propose to focus on data-centricity as the overarching design pattern which enables smarter ML system deployment and operation at scale.