mlsanityData to 7 Oct 2026

Mell: Memory-Efficient Large Language Model Serving via Multi-GPU KV Cache Management

arXiv 2501.06709 ↗First posted 12 Jan 2025cs.DCcited by 7

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Serving large language models (LLMs) for massive users is challenged by the significant memory footprint of the transient state, known as the key-value (KV) cache, which scales with sequence length and number of requests. Instead of renting or buying more expensive GPUs, the load imbalance of the KV cache across GPUs, coupled with recent advances in inter-GPU communication, provides an opportunity to serve more requests via request migration. However, high migration overhead and unpredictable request patterns make it challenging. Therefore, this paper proposes MELL, a memory-efficient LLM serving system via multi-GPU KV cache management. It saves the number of GPUs needed in the system by considering the dynamic KV cache load and the costly request migration. Specifically, we first develop an adaptive request migration mechanism to balance the computational and communication overheads and adapt to diverse resource conditions. Then, we design an online algorithm tailored to a multi-LLM request and multi-GPU scheduling problem with migration enabled. It aims to minimise the required GPUs while limiting the number of migrations. Finally, we implement a prototype of MELL and demonstrate that it reduces the number of GPUs by 31% and increases the GPU utilization by 43% at most compared to existing LLM serving systems.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 30 of its 41 references that are in the map. First posted on arXiv 12 Jan 2025.

Builds on

30 of 41 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  4. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  9. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  10. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  11. 2022Efficiently Scaling Transformer Inferencecited by 529
  12. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
14 more
  1. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  2. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  4. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  5. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  6. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  7. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 124
  8. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  9. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  10. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  11. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  12. 2024Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inferencecited by 25
  13. 2024OTAS: An Elastic Transformer Serving System via Token Adaptationcited by 3
  14. 2014Efficient Online Strategies for Renting Servers in the Cloudcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 23
  2. 2026PPAI: Enabling Personalized LLM Agent Interoperability for Collaborative Edge Intelligencecited by 3
  3. 2026ForkKV: Scaling Multi-LoRA Agent Serving via Copy-on-Write Disaggregated KV Cachecited by 2
  4. 2026Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agentscited by 0
  5. 2026MATCH: Modulating Attention via In-Context Retrieval for Long-Context Transformerscited by 0
  6. 2026Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUscited by 0
  7. 2026Crossflow: Prefill-Decode Elasticity for Agentic LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
  2. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving
  3. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
  4. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
  5. 2024FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines
  6. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  7. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storage
  8. 2025Strata: Hierarchical Context Caching for Long Context Language Model Serving