mlsanityData to 7 Oct 2026

Efficient Memory Management for Large Language Model Serving with PagedAttention

arXiv 2309.06180 ↗First posted 12 Sep 2023cs.LG cs.DCcited by 5,767

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

High throughput serving of large language models (LLMs) requires batching sufficiently many requests at a time. However, existing systems struggle because the key-value cache (KV cache) memory for each request is huge and grows and shrinks dynamically. When managed inefficiently, this memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size. To address this problem, we propose PagedAttention, an attention algorithm inspired by the classical virtual memory and paging techniques in operating systems. On top of it, we build vLLM, an LLM serving system that achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV cache within and across requests to further reduce memory usage. Our evaluations show that vLLM improves the throughput of popular LLMs by 2-4× with the same level of latency compared to the state-of-the-art systems, such as FasterTransformer and Orca. The improvement is more pronounced with longer sequences, larger models, and more complex decoding algorithms. vLLM's source code is publicly available at https://github.com/vllm-project/vllm

Key facts. Cited by 5,767 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 28 of its 62 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

28 of 62 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2016Layer Normalizationcited by 5,579
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
16 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  6. 2022Efficiently Scaling Transformer Inferencecited by 529
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  8. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  9. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  10. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 119
  11. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
  12. 2017TensorFlow-Serving: Flexible, High-Performance ML Servingcited by 85
  13. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  14. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  15. 2022POET: Training Neural Networks on Tiny Devices with Integrated Rematerialization and Pagingcited by 26
  16. 2022OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networkscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 78; 2024: 859; 2025: 2001; 2026: 2829.

5,767 papers.

Sort:

By area:

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2023Mistral 7Bcited by 3,065
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  9. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  10. 2025s1: Simple test-time scalingcited by 1,311
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
38 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  4. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  7. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  8. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  9. 2024Yi: Open Foundation Models by 01.AIcited by 706
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  11. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  12. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  13. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  14. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  15. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  16. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  17. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  18. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  19. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  20. 2024Long Context Transfer from Language to Visioncited by 502
  21. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  22. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  23. 2025Qwen3-Omni Technical Reportcited by 451
  24. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  25. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  26. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  27. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  28. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  29. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  30. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 319
  31. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  32. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  33. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  34. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 294
  35. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 289
  36. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  37. 2026Reinforcement Learning via Self-Distillationcited by 285
  38. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 283

50 most cited of 5,767 All 5,767 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  2. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head
  3. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  4. 2026Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention
  5. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
  6. 2024Model Tells You Where to Merge: Adaptive KV Cache Merging for LLMs on Long-Context Tasks
  7. 2025CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
  8. 2025Towards Efficient Key-Value Cache Management for Prefix Prefilling in LLM Inference