# Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning

arXiv: https://arxiv.org/abs/2505.04021 · HTML: https://mlsanity.com/paper/2505.04021/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 16 of its 59 references that are in the map.
- First posted on arXiv 6 May 2025.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 16 of 16)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention](https://mlsanity.com/paper/2405.04437/) (2024)
- [ServerlessLLM: Low-Latency Serverless Inference for Large Language Models](https://mlsanity.com/paper/2401.14351/) (2024)
- [ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference](https://mlsanity.com/paper/2404.07947/) (2024)
- [Queue management for slo-oriented large language model serving](https://mlsanity.com/paper/2407.00047/) (2024)
- [SLOs-Serve: Optimized Serving of Multi-SLO LLMs](https://mlsanity.com/paper/2504.08784/) (2025)
- [Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applications](https://mlsanity.com/paper/1902.04610/) (2019)
- [vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving](https://mlsanity.com/paper/2407.15309/) (2024)
- [LithOS: An Operating System for Efficient Machine Learning on GPUs](https://mlsanity.com/paper/2504.15465/) (2025)
- [Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management](https://mlsanity.com/paper/2501.06709/) (2025)
- [MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusters](https://mlsanity.com/paper/2303.13803/) (2023)
- [SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUs](https://mlsanity.com/paper/2407.13996/) (2024)
- [Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloads](https://mlsanity.com/paper/2410.07381/) (2024)
- [A simple proof of the Moore-Hodgson Algorithm for minimizing the number of late jobs](https://mlsanity.com/paper/2104.06210/) (2021)

## Built on by (most cited first, 24 of 24)

- [Barbarians at the Gate: How AI is Upending Systems Research](https://mlsanity.com/paper/2510.06189/) (2025)
- [FairBatching: Fairness-Aware Batch Formation for LLM Inference](https://mlsanity.com/paper/2510.14392/) (2025)
- [Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Serving](https://mlsanity.com/paper/2604.25899/) (2026)
- [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](https://mlsanity.com/paper/2512.09472/) (2025)
- [FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Serving](https://mlsanity.com/paper/2509.06261/) (2025)
- [xLLM Technical Report](https://mlsanity.com/paper/2510.14686/) (2025)
- [Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristics](https://mlsanity.com/paper/2508.18736/) (2025)
- [Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity](https://mlsanity.com/paper/2512.01357/) (2025)
- [Let the Barbarians In: How AI Can Accelerate Systems Performance Research](https://mlsanity.com/paper/2512.14806/) (2025)
- [ContextEvolve: Multi-Agent Context Compression for Systems Code Optimization](https://mlsanity.com/paper/2602.02597/) (2026)
- [Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Start](https://mlsanity.com/paper/2604.06664/) (2026)
- [Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs](https://mlsanity.com/paper/2605.04357/) (2026)
- [ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Serving](https://mlsanity.com/paper/2505.14468/) (2025)
- [Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelines](https://mlsanity.com/paper/2604.15186/) (2026)
- [JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training](https://mlsanity.com/paper/2604.23838/) (2026)
- [TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AI](https://mlsanity.com/paper/2607.22465/) (2026)
- [From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Models](https://mlsanity.com/paper/2511.02248/) (2025)
- [Lodestar: An Online-Learning LLM Inference Router](https://mlsanity.com/paper/2606.00946/) (2026)
- [FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-location](https://mlsanity.com/paper/2606.04415/) (2026)
- [Breaking the Ice: Analyzing Cold Start Latency in vLLM](https://mlsanity.com/paper/2606.07362/) (2026)
- [Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RAN](https://mlsanity.com/paper/2608.16477/) (2026)
- [Dynamic HBM Repartitioning for Multi-Turn MoE Serving](https://mlsanity.com/paper/2609.13537/) (2026)
- [Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamo](https://mlsanity.com/paper/2609.25451/) (2026)
- [FastEEP: Fast Intra-Instance Elastic Scaling for Expert-Parallel MoE Serving](https://mlsanity.com/paper/2610.18451/) (2026)

## Abstract (arXiv)

Inference providers must maintain availability for many LLMs, including low-volume but essential models, making resource efficiency increasingly important as token prices fall. Analysis of production traces reveals a dynamic bursty-group pattern in which sets of models become active together and shift over time; existing space- and time-sharing approaches lack principled mechanisms to adapt to this variability, forcing trade-offs between SLO adherence and efficiency. We observe that elastic memory allocation can unify spatial and temporal sharing. Based on this insight, we have developed Prism, a memory-centric LLM co-serving framework that applies memory ballooning to reclaim memory across models and support both forms of sharing under a single scheme. Prism's balloon driver, referred to as kvcached, has been open-sourced at https://github.com/ovg-project/kvcached, and deployed in production environments across 10K+ GPUs.