mlsanityData to 14 Oct 2026

Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving

arXiv 2407.00079 ↗First posted 24 Jun 2024cs.DC cs.AI cs.ARcited by 196

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake excels in long-context scenarios. Compared to the baseline method, Mooncake can achieve up to a 525% increase in throughput in certain simulated scenarios while adhering to SLOs. Under real workloads, Mooncake's innovative architecture enables Kimi to handle 75% more requests.

Key facts. Cited by 196 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 39 of its 77 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

39 of 77 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Preble: Efficient Distributed Prompt Scheduling for LLM Serving8 May
  2. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  5. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 34

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  7. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  8. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  9. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  10. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
22 more
  1. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  2. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  3. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  4. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  5. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  6. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  7. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 203
  8. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  9. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  10. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 147
  11. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attentioncited by 131
  12. 2021Sequence Parallelism: Long Sequence Training from System Perspectivecited by 126
  13. 2024Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inferencecited by 126
  14. 2024MiniCache: KV Cache Compression in Depth Dimension for Large Language Modelscited by 125
  15. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  16. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  17. 2021TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Modelscited by 75
  18. 2024Attention Score is not All You Need for Token Importance Indicator in KV Cache Reduction: Value Also Matterscited by 73
  19. 2024USP: A Unified Sequence Parallelism Approach for Long Context Generative AIcited by 61
  20. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
  21. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 54
  22. 2023X-Former: In-Memory Acceleration of Transformerscited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 63; 2026: 116.

196 papers.

Sort:

By area:

  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  2. 2025Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Modelcited by 168
  3. 2026Kimi K3: Open Frontier Intelligencecited by 138
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  5. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 97
  6. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 61
  7. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  8. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 40
  9. 2024Block-Attention for Efficient Prefillingcited by 37
  10. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 34
  11. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
  12. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
38 more
  1. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  2. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Servingcited by 23
  3. 2025KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reusecited by 22
  4. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  5. 2025Auditing Prompt Caching in Language Model APIscited by 20
  6. 2025Past-Future Scheduler for LLM Serving under SLA Guaranteescited by 20
  7. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  8. 2025Towards AI Search Paradigmcited by 19
  9. 2025ModServe: Modality- and Stage-Aware Resource Disaggregation for Scalable Multimodal Model Servingcited by 18
  10. 2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
  11. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 17
  12. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Managementcited by 16
  13. 2024Efficiently Serving Large Multimodal Models Using EPD Disaggregationcited by 16
  14. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
  15. 2025semi-PD: Towards Efficient LLM Serving via Phase-Wise Disaggregated Computation and Unified Storagecited by 16
  16. 2025StreamingThinker: Large Language Models Can Think While Readingcited by 15
  17. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 14
  18. 2025Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregationcited by 14
  19. 2025PDTrim: Targeted Pruning for Prefill-Decode Disaggregation in Inferencecited by 14
  20. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  21. 2024No Request Left Behind: Tackling Heterogeneity in Long-Context LLM Inference with Medhacited by 13
  22. 2025Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Servingcited by 13
  23. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 12
  24. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 11
  25. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
  26. 2025HACK: Homomorphic Acceleration via Compression of the Key-Value Cache for Disaggregated LLM Inferencecited by 10
  27. 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Schedulingcited by 10
  28. 2025On Evaluating Performance of LLM Inference Serving Systemscited by 10
  29. 2025IAM: Efficient Inference through Attention Mapping between Different-scale LLMscited by 10
  30. 2026AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Servingcited by 9
  31. 2024KVDirect: Distributed Disaggregated LLM Inferencecited by 8
  32. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  33. 2026Heddle: A Distributed Orchestration System for Agentic RL Rolloutcited by 8
  34. 2026TokenDance: Scaling Multi-Agent LLM Serving via Collective KV Cache Sharingcited by 8
  35. 2024CritiPrefill: A Segment-wise Criticality-based Approach for Prefilling Acceleration in LLMscited by 7
  36. 2025A2ATS: Retrieval-Based KV Cache Reduction via Windowed Rotary Position Embedding and Query-Aware Vector Quantizationcited by 7
  37. 2025ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelismcited by 7
  38. 2025Efficient Speculative Decoding for Llama at Scale: Challenges and Solutionscited by 7

50 most cited of 196 All 196 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration
  2. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demands
  3. 2024Compute Or Load KV Cache? Why Not Both?
  4. 2025Ascendra: Dynamic Request Prioritization for Efficient LLM Serving
  5. 2026Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism
  6. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving
  7. 2026KVServe: Service-Aware KV Cache Compression for Communication-Efficient Disaggregated LLM Serving
  8. 2026DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory