# SERFLOW: A Cross-Service Cost Optimization Framework for SLO-Aware Dynamic ML Inference

arXiv: https://arxiv.org/abs/2510.27182 · HTML: https://mlsanity.com/paper/2510.27182/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 27 references that are in the map.
- First posted on arXiv 31 Oct 2025.

## Builds on (most cited first, 10 of 10)

- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [BASE Layers: Simplifying Training of Large, Sparse Models](https://mlsanity.com/paper/2103.16716/) (2021)
- [Hard Mixtures of Experts for Large Scale Weakly Supervised Vision](https://mlsanity.com/paper/1704.06363/) (2017)
- [Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider](https://mlsanity.com/paper/2003.03423/) (2020)
- [Learning to Decode Collaboratively with Multiple Language Models](https://mlsanity.com/paper/2403.03870/) (2024)
- [Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems](https://mlsanity.com/paper/2502.04510/) (2025)
- [SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimization](https://mlsanity.com/paper/2410.10759/) (2024)
- [The server is dead, long live the server: Rise of Serverless Computing, Overview of Current State and Future Trends in Research and Industry](https://mlsanity.com/paper/1906.02888/) (2019)
- [Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques](https://mlsanity.com/paper/2506.06579/) (2025)
- [LIBRA: An Economical Hybrid Approach for Cloud Applications with Strict SLAs](https://mlsanity.com/paper/2104.05491/) (2021)

## Abstract (arXiv)

Dynamic offloading of Machine Learning (ML) model partitions across different resource orchestration services, such as Function-as-a-Service (FaaS) and Infrastructure-as-a-Service (IaaS), can balance processing and transmission delays while minimizing costs of adaptive inference applications. However, prior work often overlooks real-world factors, such as Virtual Machine (VM) cold starts, requests under long-tail service time distributions, etc. To tackle these limitations, we model each ML query (request) as traversing an acyclic sequence of stages, wherein each stage constitutes a contiguous block of sparse model parameters ending in an internal or final classifier where requests may exit. Since input-dependent exit rates vary, no single resource configuration suits all query distributions. IaaS-based VMs become underutilized when many requests exit early, yet rapidly scaling to handle request bursts reaching deep layers is impractical. SERFLOW addresses this challenge by leveraging FaaS-based serverless functions (containers) and using stage-specific resource provisioning that accounts for the fraction of requests exiting at each stage. By integrating this provisioning with adaptive load balancing across VMs and serverless functions based on request ingestion, SERFLOW reduces cloud costs by over 23% while efficiently adapting to dynamic workloads.