mlsanityData to 7 Oct 2026

vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

arXiv 2405.04437 ↗First posted 7 May 2024cs.LG cs.OScited by 93

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

PagedAttention is a popular approach for dynamic memory allocation in LLM serving systems. It enables on-demand allocation of GPU memory to mitigate KV cache fragmentation -- a phenomenon that crippled the batch size (and consequently throughput) in prior systems. However, in trying to allocate physical memory at runtime, PagedAttention ends up changing the virtual memory layout of the KV cache from contiguous to non-contiguous. Such a design leads to non-trivial programming and performance overheads. We present vAttention -- an approach that mitigates fragmentation in physical memory while retaining the contiguity of KV cache in virtual memory. We achieve this by decoupling the allocation of virtual and physical memory using CUDA virtual memory management APIs. We also introduce various LLM-specific optimizations to address the limitations of CUDA virtual memory support. Overall, vAttention is a simpler, portable, and performant alternative to PagedAttention: it supports various attention kernels out-of-the-box and improves LLM serving throughput by up to 1.23x compared to the use of PagedAttention-based kernels of FlashAttention and FlashInfer.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 22 of its 80 references that are in the map. First posted on arXiv 7 May 2024.

Builds on

22 of 80 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  2. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  3. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills31 Aug
  4. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  12. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
6 more
  1. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 120
  2. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  3. 2024D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Servingcited by 48
  4. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 30
  5. 2024GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitchingcited by 13
  6. 2023A Case Study in CUDA Kernel Fusion: Implementing FlashAttention-2 on NVIDIA Hopper Architecture using the CUTLASS Librarycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 29; 2026: 56.

93 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  2. 2024MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decodingcited by 87
  3. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  4. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  5. 2024MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Poolcited by 67
  6. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  7. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 46
  8. 2025Cognitive Memory in Large Language Modelscited by 28
  9. 2025Jenga: Effective Memory Management for Serving LLM with Heterogeneitycited by 24
  10. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 23
  11. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 17
  12. 2025Taming the Titans: A Survey of Efficient LLM Inference Servingcited by 16
38 more
  1. 2026Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Servingcited by 16
  2. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 15
  3. 2026LLM-42: Enabling Determinism in LLM Inference with Verified Speculationcited by 14
  4. 2024vTensor: Flexible Virtual Tensor Management for Efficient LLM Servingcited by 13
  5. 2024KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Servingcited by 10
  6. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 10
  7. 2025HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Servingcited by 8
  8. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  9. 2026Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Servingcited by 7
  10. 2025Niyama : Breaking the Silos of LLM Inference Servingcited by 6
  11. 2025AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Upcited by 6
  12. 2025Position: AI Scaling: From Up to Down and Outcited by 5
  13. 2025ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughputcited by 4
  14. 2025Boosting LLM Serving through Spatial-Temporal GPU Resource Sharingcited by 4
  15. 2025eLLM: Elastic Memory Management Framework for Efficient LLM Servingcited by 4
  16. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inferencecited by 4
  17. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Servingcited by 4
  18. 2026Runtime-Certified Bounded-Error Quantized Attentioncited by 4
  19. 2026What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agentscited by 4
  20. 2025FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Servingcited by 3
  21. 2025ElasticMoE: An Efficient Auto Scaling Method for Mixture-of-Experts Modelscited by 3
  22. 2025End-to-End Multi-Modal Diffusion Mambacited by 3
  23. 2025xLLM Technical Reportcited by 3
  24. 2026CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inferencecited by 3
  25. 2026PipeLive: Efficient Live In-place Pipeline Parallelism Reconfiguration for Dynamic LLM Servingcited by 3
  26. 2025Rethinking Caching for LLM Serving Systems: Beyond Traditional Heuristicscited by 2
  27. 2025VoltanaLLM: Energy-Efficient and SLO-Aware Disaggregated LLM Serving via Adaptive Frequency Control and State-Space Routingcited by 2
  28. 2025WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Servingcited by 2
  29. 2025Dynamic Rebatching for Efficient Early-Exit Inference with DREXcited by 2
  30. 2026Mosaic: Unlocking Long-Context Inference for Diffusion LLMs via Global Memory Planning and Dynamic Peak Tamingcited by 2
  31. 2026Dynamic Sparse Attention: Access Patterns and Architecturecited by 2
  32. 2026HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computingcited by 2
  33. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  34. 2026SMetric: Rethink LLM Scheduling for Serving Agents with Balanced Session-centric Schedulingcited by 2
  35. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  36. 2024Understanding and Alleviating Memory Consumption in RLHF for LLMscited by 1
  37. 2025iServe: An Intent-based Serving System for LLMscited by 1
  38. 2025Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inferencecited by 1

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash
  2. 2026OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration
  3. 2026Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
  4. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  5. 2026Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management
  6. 2026SLIM: Saturation-Aware Lightweight Performance Modeling for LLM Serving
  7. 2026A CXL Memory Rack for Multi-Turn LLM Serving
  8. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention