mlsanityData to 13 Oct 2026

Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning

arXiv 2505.04021 ↗First posted 6 May 2025cs.DC cs.AI cs.LGcited by 23

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Inference providers must maintain availability for many LLMs, including low-volume but essential models, making resource efficiency increasingly important as token prices fall. Analysis of production traces reveals a dynamic bursty-group pattern in which sets of models become active together and shift over time; existing space- and time-sharing approaches lack principled mechanisms to adapt to this variability, forcing trade-offs between SLO adherence and efficiency. We observe that elastic memory allocation can unify spatial and temporal sharing. Based on this insight, we have developed Prism, a memory-centric LLM co-serving framework that applies memory ballooning to reclaim memory across models and support both forms of sharing under a single scheme. Prism's balloon driver, referred to as kvcached, has been open-sourced at https://github.com/ovg-project/kvcached, and deployed in production environments across 10K+ GPUs.

Key facts. Cited by 23 papers in the map (data through 13 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 16 of its 59 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

16 of 59 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management12 Jan
  2. 2024Queue management for slo-oriented large language model serving5 Jun
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  4. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 11

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  2. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  3. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  4. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 18
  5. 2019Salus: Fine-Grained GPU Sharing Primitives for Deep Learning Applicationscited by 17
  6. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Servingcited by 14
  7. 2025LithOS: An Operating System for Efficient Machine Learning on GPUscited by 9
  8. 2023MuxFlow: Efficient and Safe GPU Sharing in Large-Scale Production Deep Learning Clusterscited by 7
  9. 2024SGDRC: Software-Defined Dynamic Resource Control for Concurrent DNN Inference on NVIDIA GPUscited by 7
  10. 2024Tally: Non-Intrusive Performance Isolation for Concurrent Deep Learning Workloadscited by 7
  11. 2021A simple proof of the Moore-Hodgson Algorithm for minimizing the number of late jobscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 10; 2026: 13.

23 papers.

Sort:

By area:

  1. 2025Barbarians at the Gate: How AI is Upending Systems Researchcited by 50
  2. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  3. 2026Pythia: Exploiting Workflow Predictability for Efficient Agent-Native LLM Servingcited by 6
  4. 2025FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Servingcited by 3
  5. 2025xLLM Technical Reportcited by 3
  6. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristicscited by 2
  7. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 2
  8. 2025Let the Barbarians In: How AI Can Accelerate Systems Performance Researchcited by 2
  9. 2026ContextEvolve: Multi-Agent Context Compression for Systems Code Optimizationcited by 2
  10. 2025ServerlessLoRA: Enabling Low-Latency Serverless Multi-LoRA Servingcited by 1
  11. 2025Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinitycited by 1
  12. 2026Foundry: Template-Based CUDA Graph Context Materialization for Fast LLM Serving Cold Startcited by 1
11 more
  1. 2026Scepsy: Serving Agentic Workflows Using Aggregate LLM Pipelinescited by 1
  2. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 1
  3. 2026Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUscited by 1
  4. 2026TRACE-ROUTER: Task-Consistent and Adaptive Online Routing for Agentic AIcited by 1
  5. 2025From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Modelscited by 0
  6. 2026Lodestar: An Online-Learning LLM Inference Routercited by 0
  7. 2026FlexNPU: Transparent NPU Virtualization for Dynamic LLM Prefill-Decode Co-locationcited by 0
  8. 2026Breaking the Ice: Analyzing Cold Start Latency in vLLMcited by 0
  9. 2026Pallas: A Proactive KV Cache Migration Framework for LLM Inference in AI-RANcited by 0
  10. 2026Dynamic HBM Repartitioning for Multi-Turn MoE Servingcited by 0
  11. 2026Fast Recovery for LLM Serving via Decoupled Device Memory Lifetime in Dynamocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026LLMVisor: A Real-Time Latency Attribution Model for Multi-Tenant LLM Serving
  2. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  3. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  4. 2024Saving GPU Hours in LLM Inference System Development and Online Workloads with Simulation and DBMS-Inspired Cache Replacement Policies
  5. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  6. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  7. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  8. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains