# vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention

arXiv: https://arxiv.org/abs/2405.04437 · HTML: https://mlsanity.com/paper/2405.04437/

## Key facts

- Cited by 98 papers in the map (data through 14 Oct 2026).
- Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving.
- Builds on 22 of its 80 references that are in the map.
- First posted on arXiv 7 May 2024.

## Placed in

- Efficient LLM inference and serving / [Serving systems and request scheduling](https://mlsanity.com/inference/serving-systems-and-request-scheduling/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [SGLang: Efficient Execution of Structured Language Model Programs](https://mlsanity.com/paper/2312.07104/) (2023)
- [Fast Transformer Decoding: One Write-Head is All You Need](https://mlsanity.com/paper/1911.02150/) (2019)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills](https://mlsanity.com/paper/2308.16369/) (2023)
- [Fast Distributed Inference Serving for Large Language Models](https://mlsanity.com/paper/2305.05920/) (2023)
- [Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads](https://mlsanity.com/paper/2401.11181/) (2024)
- [DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inference](https://mlsanity.com/paper/2401.08671/) (2024)
- [D'ej`aVu: KV-cache Streaming for Fast, Fault-tolerant Generative LLM Serving](https://mlsanity.com/paper/2403.01876/) (2024)
- [M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity](https://mlsanity.com/paper/2404.14527/) (2024)
- [GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching](https://mlsanity.com/paper/2401.08156/) (2024)
- [A Case Study in CUDA Kernel Fusion: Implementing FlashAttention-2 on NVIDIA Hopper Architecture using the CUTLASS Library](https://mlsanity.com/paper/2312.11918/) (2023)

## Built on by (most cited first, 30 of 98)

- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [MagicDec: Breaking the Latency-Throughput Tradeoff for Long Context Generation with Speculative Decoding](https://mlsanity.com/paper/2408.11049/) (2024)
- [A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression](https://mlsanity.com/paper/2406.11430/) (2024)
- [Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution](https://mlsanity.com/paper/2510.00636/) (2025)
- [MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool](https://mlsanity.com/paper/2406.17565/) (2024)
- [POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inference](https://mlsanity.com/paper/2410.18038/) (2024)
- [LLM Inference Serving: Survey of Recent Advances and Opportunities](https://mlsanity.com/paper/2407.12391/) (2024)
- [Cognitive Memory in Large Language Models](https://mlsanity.com/paper/2504.02441/) (2025)
- [Jenga: Effective Memory Management for Serving LLM with Heterogeneity](https://mlsanity.com/paper/2503.18292/) (2025)
- [Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning](https://mlsanity.com/paper/2505.04021/) (2025)
- [Parallelizing Tool Execution and LLM Generation for Low-Latency Agent Serving](https://mlsanity.com/paper/2603.18897/) (2026)
- [SLOs-Serve: Optimized Serving of Multi-SLO LLMs](https://mlsanity.com/paper/2504.08784/) (2025)
- [Taming the Titans: A Survey of Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.19720/) (2025)
- [LLM-42: Enabling Determinism in LLM Inference with Verified Speculation](https://mlsanity.com/paper/2601.17768/) (2026)
- [vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving](https://mlsanity.com/paper/2407.15309/) (2024)
- [Pie: A Programmable Serving System for Emerging LLM Applications](https://mlsanity.com/paper/2510.24051/) (2025)
- [KunServe: Parameter-centric Memory Management for Efficient Memory Overloading Handling in LLM Serving](https://mlsanity.com/paper/2412.18169/) (2024)
- [Efficient LLM Serving for Agentic Workflows: A Data Systems Perspective](https://mlsanity.com/paper/2603.16104/) (2026)
- [HELIOS: Adaptive Model And Early-Exit Selection for Efficient LLM Inference Serving](https://mlsanity.com/paper/2504.10724/) (2025)
- [M2XFP: A Metadata-Augmented Microscaling Data Format for Efficient Low-bit Quantization](https://mlsanity.com/paper/2601.19213/) (2026)
- [DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving](https://mlsanity.com/paper/2602.06502/) (2026)
- [Niyama : Breaking the Silos of LLM Inference Serving](https://mlsanity.com/paper/2503.22562/) (2025)
- [Execution-State Capsules: Graph-Bound Execution-State Checkpoint and Restore for Low-Latency, Small-Batch, On-Device Physical-AI Serving](https://mlsanity.com/paper/2606.20537/) (2026)
- [AutoChemSchematic AI: Agentic Physics-Aware Automation for Chemical Manufacturing Scale-Up](https://mlsanity.com/paper/2505.24584/) (2025)
- [Position: AI Scaling: From Up to Down and Out](https://mlsanity.com/paper/2502.01677/) (2025)
- [eLLM: Elastic Memory Management Framework for Efficient LLM Serving](https://mlsanity.com/paper/2506.15155/) (2025)
- [WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving](https://mlsanity.com/paper/2512.09472/) (2025)
- [CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inference](https://mlsanity.com/paper/2604.06036/) (2026)
- [ADOR: A Design Exploration Framework for LLM Serving with Enhanced Latency and Throughput](https://mlsanity.com/paper/2503.04253/) (2025)
- [Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing](https://mlsanity.com/paper/2504.19516/) (2025)

## Abstract (arXiv)

PagedAttention is a popular approach for dynamic memory allocation in LLM serving systems. It enables on-demand allocation of GPU memory to mitigate KV cache fragmentation -- a phenomenon that crippled the batch size (and consequently throughput) in prior systems. However, in trying to allocate physical memory at runtime, PagedAttention ends up changing the virtual memory layout of the KV cache from contiguous to non-contiguous. Such a design leads to non-trivial programming and performance overheads. We present vAttention -- an approach that mitigates fragmentation in physical memory while retaining the contiguity of KV cache in virtual memory. We achieve this by decoupling the allocation of virtual and physical memory using CUDA virtual memory management APIs. We also introduce various LLM-specific optimizations to address the limitations of CUDA virtual memory support. Overall, vAttention is a simpler, portable, and performant alternative to PagedAttention: it supports various attention kernels out-of-the-box and improves LLM serving throughput by up to 1.23x compared to the use of PagedAttention-based kernels of FlashAttention and FlashInfer.