# Keep Your Friends Close: Leveraging Affinity Groups to Accelerate AI Inference Workflows

arXiv: https://arxiv.org/abs/2312.11488 · HTML: https://mlsanity.com/paper/2312.11488/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 42 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 11 of 11)

- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Clipper: A Low-Latency Online Prediction Serving System](https://mlsanity.com/paper/1612.03079/) (2016)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems](https://mlsanity.com/paper/1810.06115/) (2018)
- [End-to-end Optimization of Machine Learning Prediction Queries](https://mlsanity.com/paper/2206.00136/) (2022)
- [Machine Learning at Microsoft with ML .NET](https://mlsanity.com/paper/1905.05715/) (2019)
- [Real-Time Machine Learning: The Missing Pieces](https://mlsanity.com/paper/1703.03924/) (2017)
- [Following the Data, Not the Function: Rethinking Function Orchestration in Serverless Computing](https://mlsanity.com/paper/2109.13492/) (2021)
- [Declarative Data Analytics: a Survey](https://mlsanity.com/paper/1902.01304/) (2019)
- [Cloudburst: Stateful Functions-as-a-Service](https://mlsanity.com/paper/2001.04592/) (2020)
- [Cascade: A Platform for Delay-Sensitive Edge Intelligence](https://mlsanity.com/paper/2311.17329/) (2023)

## Abstract (arXiv)

AI inference workflows are typically structured as a pipeline or graph of AI programs triggered by events. As events occur, the AIs perform inference or classification tasks under time pressure to respond or take some action. Standard techniques that reduce latency in other streaming settings (such as caching and optimization-driven scheduling) are of limited value because AI data access patterns (models, databases) change depending on the triggering event: a significant departure from traditional streaming. In this work, we propose a novel affinity grouping mechanism that makes it easier for developers to express application-specific data access correlations, enabling coordinated management of data objects in server clusters hosting streaming inference tasks. Our proposals are thus complementary to other approaches such as caching and scheduling. Experiments confirm the limitations of standard techniques, while showing that the proposed mechanism is able to maintain significantly lower latency as workload and scale-out increase, and yet requires only minor code changes.