# Biathlon: Harnessing Model Resilience for Accelerating ML Inference Pipelines

arXiv: https://arxiv.org/abs/2405.11191 · HTML: https://mlsanity.com/paper/2405.11191/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 89 references that are in the map.
- First posted on arXiv 18 May 2024.

## Builds on (most cited first, 20 of 20)

- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Why do tree-based models still outperform deep learning on tabular data?](https://mlsanity.com/paper/2207.08815/) (2022)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [Feature Importance-aware Transferable Adversarial Attacks](https://mlsanity.com/paper/2107.14185/) (2021)
- [NoScope: Optimizing Neural Network Queries over Video at Scale](https://mlsanity.com/paper/1703.02529/) (2017)
- [Rafiki: Machine Learning as an Analytics Service System](https://mlsanity.com/paper/1804.06087/) (2018)
- [PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems](https://mlsanity.com/paper/1810.06115/) (2018)
- [A Tensor Compiler for Unified Machine Learning Prediction Serving](https://mlsanity.com/paper/2010.04804/) (2020)
- [End-to-end Optimization of Machine Learning Prediction Queries](https://mlsanity.com/paper/2206.00136/) (2022)
- [Query Processing on Tensor Computation Runtimes](https://mlsanity.com/paper/2203.01877/) (2022)
- [Serving and Optimizing Machine Learning Workflows on Heterogeneous Infrastructures](https://mlsanity.com/paper/2205.04713/) (2022)
- [VerdictDB: Universalizing Approximate Query Processing](https://mlsanity.com/paper/1804.00770/) (2018)
- [Machine Learning at Microsoft with ML .NET](https://mlsanity.com/paper/1905.05715/) (2019)
- [Feature Importance Guided Attack: A Model Agnostic Adversarial Attack](https://mlsanity.com/paper/2106.14815/) (2021)
- [Interpretable Machine Learning: Moving From Mythos to Diagnostics](https://mlsanity.com/paper/2103.06254/) (2021)
- [AccMPEG: Optimizing Video Encoding for Video Analytics](https://mlsanity.com/paper/2204.12534/) (2022)
- [Data Science through the looking glass and what we found there](https://mlsanity.com/paper/1912.09536/) (2019)
- [JoinBoost: Grow Trees Over Normalized Data Using Only SQL](https://mlsanity.com/paper/2307.00422/) (2023)
- [Database Learning: Toward a Database that Becomes Smarter Every Time](https://mlsanity.com/paper/1703.05468/) (2017)
- [A Step Toward Deep Online Aggregation (Extended Version)](https://mlsanity.com/paper/2303.04103/) (2023)

## Built on by (most cited first, 2 of 2)

- [Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving](https://mlsanity.com/paper/2504.07494/) (2025)
- [Atlas: Optimizing Deployment of Compound AI Workflows on Heterogeneous Clusters](https://mlsanity.com/paper/2609.04513/) (2026)

## Abstract (arXiv)

Machine learning inference pipelines commonly encountered in data science and industries often require real-time responsiveness due to their user-facing nature. However, meeting this requirement becomes particularly challenging when certain input features require aggregating a large volume of data online. Recent literature on interpretable machine learning reveals that most machine learning models exhibit a notable degree of resilience to variations in input. This suggests that machine learning models can effectively accommodate approximate input features with minimal discernible impact on accuracy. In this paper, we introduce Biathlon, a novel ML serving system that leverages the inherent resilience of models and determines the optimal degree of approximation for each aggregation feature. This approach enables maximum speedup while ensuring a guaranteed bound on accuracy loss. We evaluate Biathlon on real pipelines from both industry applications and data science competitions, demonstrating its ability to meet real-time latency requirements by achieving 5.3x to 16.6x speedup with almost no accuracy loss.