mlsanityData to 7 Oct 2026

ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition

arXiv 2402.15220 ↗First posted 23 Feb 2024cs.LG cs.CLcited by 81

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Self-attention is an essential component of large language models (LLM) but a significant source of inference latency for long sequences. In multi-tenant LLM serving scenarios, the compute and memory operation cost of self-attention can be optimized by using the probability that multiple LLM requests have shared system prompts in prefixes. In this paper, we introduce ChunkAttention, a prefix-aware self-attention module that can detect matching prompt prefixes across multiple requests and share their key/value tensors in memory at runtime to improve the memory utilization of KV cache. This is achieved by breaking monolithic key/value tensors into smaller chunks and structuring them into the auxiliary prefix tree. Consequently, on top of the prefix-tree based KV cache, we design an efficient self-attention kernel, where a two-phase partition algorithm is implemented to improve the data locality during self-attention computation in the presence of shared system prompts. Experiments show that ChunkAttention can speed up the self-attention kernel by 3.2-4.8× compared to the state-of-the-art implementation, with the length of the system prompt ranging from 1024 to 4096.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 42 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

21 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep
  2. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughput9 Jun

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  6. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2023PaLM 2 Technical Reportcited by 1,154
  10. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  11. 2022A Survey on In-context Learningcited by 717
  12. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
7 more
  1. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  2. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 403
  3. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 347
  4. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 273
  5. 2018Online normalizer calculation for softmaxcited by 154
  6. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  7. 2020E-BATCH: Energy-Efficient and High-Throughput RNN Batchingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 27; 2026: 39.

81 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  2. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  5. 2024RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generationcited by 97
  6. 2024Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumptioncited by 96
  7. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoningcited by 96
  8. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  9. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  10. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  11. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  12. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 31
38 more
  1. 2025Cognitive Memory in Large Language Modelscited by 28
  2. 2024P/D-Serve: Serving Disaggregated Large Language Model at Scalecited by 27
  3. 2024DeFT: Decoding with Flash Tree-attention for Efficient Tree-structured LLM Inferencecited by 19
  4. 2025Auditing Prompt Caching in Language Model APIscited by 19
  5. 2025Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inferencecited by 18
  6. 2024Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionariescited by 17
  7. 2025Pie: A Programmable Serving System for Emerging LLM Applicationscited by 15
  8. 2026TraceLab: Characterizing Coding Agent Workloads for LLM Servingcited by 14
  9. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 13
  10. 2026CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Controlcited by 13
  11. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationcited by 11
  12. 2025TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Schedulingcited by 10
  13. 2026Efficient LLM Serving for Agentic Workflows: A Data Systems Perspectivecited by 10
  14. 2025Faster MoE LLM Inference for Extremely Large Modelscited by 9
  15. 2026PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cachecited by 9
  16. 2026DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Servingcited by 8
  17. 2026HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Cachingcited by 7
  18. 2024InstCache: A Predictive Cache for LLM Servingcited by 6
  19. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Timecited by 5
  20. 2025DAM: Dynamic Attention Mask for Long-Context Large Language Model Inference Accelerationcited by 5
  21. 2025TokenLake: A Unified Segment-level Prefix Cache Pool for Fine-grained Elastic Long-Context LLM Servingcited by 5
  22. 2025PAT: Accelerating LLM Decoding via Prefix-Aware Attention with Resource Efficient Multi-Tile Kernelcited by 5
  23. 2025CoDec: Prefix-Shared Decoding Kernel for LLMscited by 4
  24. 2025A Survey of LLM × DATAcited by 4
  25. 2026Sparse Prefix Caching for Hybrid and Recurrent LLM Servingcited by 4
  26. 2026WiCER: Wiki-memory Compile, Evaluate, Refine Iterative Knowledge Compilation for LLM Wiki Systemscited by 4
  27. 2026KV Cache Translation across Heterogeneous Large Language Modelscited by 4
  28. 2024Dynamic Activation Pitfalls in LLaMA Models: An Empirical Studycited by 3
  29. 2025ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Cachingcited by 3
  30. 2025Generative Caching for Structurally Similar Prompts and Responsescited by 3
  31. 2026CodecSight: Leveraging Video Codec Signals for Efficient Streaming VLM Inferencecited by 3
  32. 2026DepthKV: Layer-Dependent KV Cache Pruning for Long-Context LLM Inferencecited by 3
  33. 2026MemDecay: Region-Aware KV Cache Eviction for Efficient LLM Agent Inferencecited by 3
  34. 2025AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cachecited by 2
  35. 2026RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inferencecited by 2
  36. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 2
  37. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 2
  38. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
  2. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  3. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
  4. 2026Unifying Sparse Attention with Hierarchical Memory for Scalable Long-Context LLM Serving
  5. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  6. 2026Self-Indexing KVCache: Predicting Sparse Attention from Compressed Keys
  7. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs
  8. 2025Paged Attention Meets FlexAttention: Unlocking Long-Context Efficiency in Deployed Inference