mlsanityData to 14 Oct 2026

MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool

arXiv 2406.17565 ↗First posted 25 Jun 2024cs.DCcited by 68

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model (LLM) serving has transformed from stateless to stateful systems, utilizing techniques like context caching and disaggregated inference. These optimizations extend the lifespan and domain of the KV cache, necessitating a new architectural approach. We present MemServe, a unified system that integrates both inter-request and intra-request optimizations. MemServe introduces MemPool, an elastic memory pool managing distributed memory and KV caches across serving instances. Using MemPool APIs, MemServe combines context caching with disaggregated inference for the first time, supported by a global scheduler that enhances cache reuse through a global prompt tree-based locality-aware policy. Tests show that MemServe significantly improves job completion time and time-to-first-time.

Key facts. Cited by 68 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 26 of its 39 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

26 of 39 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving18 May
  2. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  3. 2024A Survey on Efficient Inference for Large Language Models22 Apr
  4. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  5. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  7. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan
  8. 2023FlashDecoding++: Faster Large Language Model Inference on GPUs2 Nov
  9. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  10. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 16

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  9. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  10. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  11. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  12. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
4 more
  1. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  2. 2023Stateful Large Language Model Serving with Pensievecited by 61
  3. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  4. 2022ByteTransformer: A High-Performance Transformer Boosted for Variable-Length Inputscited by 32

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 28; 2026: 31.

68 papers.

Sort:

By area:

  1. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  2. 2024EPIC: Efficient Position-Independent Caching for Serving Large Language Modelscited by 47
  3. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  4. 2025Measuring the environmental impact of delivering AI at Google Scalecited by 32
  5. 2025Cognitive Memory in Large Language Modelscited by 29
  6. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  7. 2025Strata: Hierarchical Context Caching for Long Context Language Model Servingcited by 26
  8. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  9. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Servingcited by 18
  10. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  11. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 16
  12. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
38 more
  1. 2025Energy Use of AI Inference, Efficiency Pathways, and Test-Time Scalingcited by 13
  2. 2025Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache Managementcited by 11
  3. 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Schedulingcited by 10
  4. 2025Beyond the Buzz: A Pragmatic Take on Inference Disaggregationcited by 9
  5. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
  6. 2025MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Servingcited by 8
  7. 2024KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Modelscited by 6
  8. 2025TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systemscited by 6
  9. 2025FairBatching: Fairness-Aware Batch Formation for LLM Inferencecited by 6
  10. 2025Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecturecited by 5
  11. 2025BucketServe: Bucket-Based Dynamic Batching for Smart and Efficient LLM Inference Servingcited by 5
  12. 2026Leyline: KV Cache Directives for Agentic Inferencecited by 5
  13. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  14. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Accelerationcited by 4
  15. 2025DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexingcited by 4
  16. 2026AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agentscited by 4
  17. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  18. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  19. 2025BanaServe: Unified KV Cache and Dynamic Module Migration for Balancing Disaggregated LLM Serving in AI Infrastructurecited by 3
  20. 2026Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stagecited by 3
  21. 2024G-RAG: Knowledge Expansion in Material Sciencecited by 2
  22. 2025DiSCo: Device-Server Collaborative LLM-Based Text Streaming Servicescited by 2
  23. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristicscited by 2
  24. 2025Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inferencecited by 2
  25. 2026RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inferencecited by 2
  26. 2026SuperInfer: SLO-Aware Rotary Scheduling and Memory Management for LLM Inference on Superchipscited by 2
  27. 2026Observation, Not Prediction: Conversation-Level Disaggregated Scheduling for Agentic Servingcited by 2
  28. 2026[AAFLOW+] Stateful Operator Abstraction with Zero-Copy Distributed KV Cache Orchestration for Multi-Agent Workflowscited by 2
  29. 2024Less is More: Optimizing Function Calling for LLM Execution on Edge Devicescited by 1
  30. 2025Reasoning Language Model Inference Serving Unveiled: An Empirical Studycited by 1
  31. 2026Efficient Remote KV Cache Reuse with GPU-native Video Codeccited by 1
  32. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inferencecited by 1
  33. 2026Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Servicecited by 1
  34. 2026Stage-Aware Communication Scheduling for Disaggregated LLM Servingcited by 1
  35. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reusecited by 1
  36. 2026TOPAS: Workflow-Aware Prefix-State Scheduling for Multi-Agent LLM Servingcited by 1
  37. 2025Boosting Embodied AI Agents through Perception-Generation Disaggregation and Asynchronous Pipeline Executioncited by 0
  38. 2025From Models to Operators: Rethinking Autoscaling Granularity for Large Generative Modelscited by 0

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving
  2. 2026PRISM: Fast Online LLM Serving via Scheduling-Memory Co-design
  3. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Serving
  4. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  5. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
  6. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  7. 2024A System for Microserving of LLMs
  8. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider