mlsanityData to 7 Oct 2026

DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads

arXiv 2410.10819 ↗First posted 14 Oct 2024cs.CLcited by 283

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Deploying long-context large language models (LLMs) is essential but poses significant computational and memory challenges. Caching all Key and Value (KV) states across all attention heads consumes substantial memory. Existing KV cache pruning methods either damage the long-context capabilities of LLMs or offer only limited efficiency improvements. In this paper, we identify that only a fraction of attention heads, a.k.a, Retrieval Heads, are critical for processing long contexts and require full attention across all tokens. In contrast, all other heads, which primarily focus on recent tokens and attention sinks--referred to as Streaming Heads--do not require full attention. Based on this insight, we introduce DuoAttention, a framework that only applies a full KV cache to retrieval heads while using a light-weight, constant-length KV cache for streaming heads, which reduces both LLM's decoding and pre-filling memory and latency without compromising its long-context abilities. DuoAttention uses a lightweight, optimization-based algorithm with synthetic data to identify retrieval heads accurately. Our method significantly reduces long-context inference memory by up to 2.55x for MHA and 1.67x for GQA models while speeding up decoding by up to 2.18x and 1.50x and accelerating pre-filling by up to 1.73x and 1.63x for MHA and GQA models, respectively, with minimal accuracy loss compared to full attention. Notably, combined with quantization, DuoAttention enables Llama-3-8B decoding with 3.3 million context length on a single A100 GPU. Code is provided in https://github.com/mit-han-lab/duo-attention.

Key facts. Cited by 283 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 49 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

33 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  2. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 29

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
17 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  3. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  6. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  10. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  11. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  12. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  13. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  14. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  15. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  16. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  17. 2023FlashDecoding++: Faster Large Language Model Inference on GPUscited by 70

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 112; 2026: 156.

283 papers.

Sort:

By area:

  1. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  2. 2025Sparse VideoGen: Accelerating Video Diffusion Transformers with Spatial-Temporal Sparsitycited by 204
  3. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 131
  4. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  6. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  7. 2025SpargeAttention: Accurate and Training-free Sparse Attention Accelerating Any Model Inferencecited by 113
  8. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  9. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  10. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  11. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  12. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
38 more
  1. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  2. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  3. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  4. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 41
  5. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compressioncited by 41
  6. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  7. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  8. 2025Sparse-vDiT: Unleashing the Power of Sparse Attention to Accelerate Video Diffusion Transformerscited by 34
  9. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 32
  10. 2025InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptationcited by 31
  11. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
  12. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  13. 2026IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reusecited by 27
  14. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  15. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
  16. 2025Accumulating Context Changes the Beliefs of Language Modelscited by 25
  17. 2025VORTA: Efficient Video Diffusion via Routing Sparse Attentioncited by 23
  18. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 23
  19. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  20. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  21. 2025DraftAttention: Fast Video Diffusion via Low-Resolution Attention Guidancecited by 21
  22. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 20
  23. 2025DiTFastAttnV2: Head-wise Attention Compression for Multi-Modality Diffusion Transformerscited by 19
  24. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 18
  25. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 18
  26. 2025PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Modelscited by 17
  27. 2025Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scorescited by 16
  28. 2025LAVa: Layer-wise KV Cache Eviction with Dynamic Budget Allocationcited by 16
  29. 2024AsymRnR: Video Diffusion Transformers Acceleration with Asymmetric Reduction and Restorationcited by 15
  30. 2024A Silver Bullet or a Compromise for Full Attention? A Comprehensive Study of Gist Token-based Context Compressioncited by 15
  31. 2025The Pitfalls of KV Cache Compressioncited by 15
  32. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  33. 2026Efficient Autoregressive Video Diffusion with Dummy Headcited by 15
  34. 2025Scaling Inference-Efficient Language Modelscited by 14
  35. 2025FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inferencecited by 14
  36. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
  37. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  38. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 14

50 most cited of 283 All 283 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference
  2. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
  3. 2024Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
  4. 2024Beyond KV Caching: Shared Attention for Efficient LLMs
  5. 2025Retrospective Sparse Attention for Efficient Long-Context Generation
  6. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
  7. 2024SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
  8. 2025SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning