# Sponge: Inference Serving with Dynamic SLOs Using In-Place Vertical Scaling

arXiv: https://arxiv.org/abs/2404.00704 · HTML: https://mlsanity.com/paper/2404.00704/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Mar 2024.

## Built on by (most cited first, 2 of 2)

- [Predictable LLM Serving on GPU Clusters](https://mlsanity.com/paper/2508.20274/) (2025)
- [A Study of Skews, Imbalances, and Pathological Conditions in LLM Inference Deployment on GPU Clusters detectable from DPU](https://mlsanity.com/paper/2509.18114/) (2025)

## Abstract (arXiv)

Mobile and IoT applications increasingly adopt deep learning inference to provide intelligence. Inference requests are typically sent to a cloud infrastructure over a wireless network that is highly variable, leading to the challenge of dynamic Service Level Objectives (SLOs) at the request level. This paper presents Sponge, a novel deep learning inference serving system that maximizes resource efficiency while guaranteeing dynamic SLOs. Sponge achieves its goal by applying in-place vertical scaling, dynamic batching, and request reordering. Specifically, we introduce an Integer Programming formulation to capture the resource allocation problem, providing a mathematical model of the relationship between latency, batch size, and resources. We demonstrate the potential of Sponge through a prototype implementation and preliminary experiments and discuss future works.