mlsanityData to 7 Oct 2026

The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving

arXiv 2405.11299 ↗First posted 18 May 2024cs.DB cs.LGcited by 3

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

We survey the large language model (LLM) serving area to understand the intricate dynamics between cost-efficiency and accuracy, which is magnified by the growing need for longer contextual understanding when deploying models at a massive scale. Our findings reveal that works in this space optimize along three distinct but conflicting goals: improving serving context length (C), improving serving accuracy (A), and improving serving performance (P). Drawing inspiration from the CAP theorem in databases, we propose a CAP principle for LLM serving, which suggests that any optimization can improve at most two of these three goals simultaneously. Our survey categorizes existing works within this framework. We find the definition and continuity of user-perceived measurement metrics are crucial in determining whether a goal has been met, akin to prior CAP databases in the wild. We recognize the CAP principle for LLM serving as a guiding principle, rather than a formal theorem, to inform designers of the inherent and dynamic trade-offs in serving models. As serving accuracy and performance have been extensively studied, this survey focuses on works that extend serving context length and address the resulting challenges.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 87 of its 106 references that are in the map. First posted on arXiv 18 May 2024.

Builds on

87 of 106 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024A Survey on Efficient Inference for Large Language Models22 Apr
  2. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelism15 Apr
  3. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCache5 Jan

Other references · 84

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  7. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  10. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  11. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  3. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  4. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  5. 2020Reformer: The Efficient Transformercited by 1,658
  6. 2020Rethinking Attention with Performerscited by 1,584
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  9. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  11. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  13. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
  14. 2022Efficiently Scaling Transformer Inferencecited by 529
  15. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  16. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  17. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  18. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  19. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  20. 2018Efficient Attention: Attention with Linear Complexitiescited by 366
  21. 2023Adapting Language Models to Compress Contextscited by 350
  22. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 343
  23. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  24. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  25. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 279
  26. 2023A Survey on Model Compression for Large Language Modelscited by 263
  27. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  28. 2020Sparse Sinkhorn Attentioncited by 250
  29. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  30. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  31. 2024Capabilities of Gemini Models in Medicinecited by 231
  32. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  33. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  34. 2022Recurrent Memory Transformercited by 197
  35. 2024Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attentioncited by 194
  36. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  37. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
  38. 2019Blockwise Self-Attention for Long Document Understandingcited by 158

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  2. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  3. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67

Similar, not cited

Close in wording, no citation either way

  1. 2024ScaleLLM: A Resource-Frugal LLM Serving Framework by Optimizing End-to-End Efficiency
  2. 2024An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks
  3. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling
  4. 2025ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production
  5. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
  6. 2025LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
  7. 2025On the Fundamental Limits of LLMs at Scale
  8. 2025IC-Cache: Efficient Large Language Model Serving via In-context Caching