mlsanityData to 7 Oct 2026

RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval

arXiv 2409.10516 ↗First posted 16 Sep 2024cs.LG cs.CLcited by 114

Not on the map. Why?

Cited by 62 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer-based Large Language Models (LLMs) have become increasingly important. However, due to the quadratic time complexity of attention computation, scaling LLMs to longer contexts incurs extremely slow inference speed and high GPU memory consumption for caching key-value (KV) vectors. This paper proposes RetrievalAttention, a training-free approach to both accelerate attention computation and reduce GPU memory consumption. By leveraging the dynamic sparsity of attention mechanism, RetrievalAttention proposes to build approximate nearest neighbor search (ANNS) indexes for KV vectors in CPU memory and retrieve the most relevant ones through vector search during generation. Unfortunately, we observe that the off-the-shelf ANNS indexes are often ineffective for such retrieval tasks due to the out-of-distribution (OOD) between query vectors and key vectors in the attention mechanism. RetrievalAttention addresses the OOD challenge by designing an attention-aware vector search algorithm that can adapt to the distribution of query vectors. Our evaluation demonstrates that RetrievalAttention achieves near full attention accuracy while only requiring access to 1--3% of the data. This leads to a significant reduction in the inference cost of long-context LLMs, with a much lower GPU memory footprint. In particular, RetrievalAttention only needs a single NVIDIA RTX4090 (24GB) to serve 128K tokens for LLMs with 8B parameters, which is capable of generating one token in 0.188 seconds.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). Builds on 44 of its 67 references that are in the map. First posted on arXiv 16 Sep 2024.

Builds on

44 of 67 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2020Reformer: The Efficient Transformercited by 1,658
  8. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  11. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
32 more
  1. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  4. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  5. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  6. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  7. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 487
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  10. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  11. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 428
  12. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  13. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  14. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  15. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  16. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 239
  17. 2022Memorizing Transformerscited by 219
  18. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  19. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  20. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  21. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 121
  22. 2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
  23. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  24. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 93
  25. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 87
  26. 2021A Comprehensive Survey and Experimental Comparison of Graph-Based Approximate Nearest Neighbor Searchcited by 82
  27. 2020PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterestcited by 75
  28. 2024RepoAgent: An LLM-Powered Open-Source Framework for Repository-level Code Documentation Generationcited by 69
  29. 2022OOD-DiskANN: Efficient and Scalable Graph ANNS for Out-of-Distribution Queriescited by 14
  30. 2024RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Searchcited by 12
  31. 2024IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUscited by 8
  32. 2024CuriousLLM: Elevating Multi-Document Question Answering with LLM-Enhanced Knowledge Graph Reasoningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 60; 2026: 47.

114 papers.

Sort:

By area:

  1. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  2. 2025MemOS: A Memory OS for AI Systemcited by 129
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 112
  5. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  6. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  7. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  8. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 48
  9. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  10. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  11. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 38
  12. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
38 more
  1. 2025O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agentscited by 30
  2. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 28
  3. 2025QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cachecited by 26
  4. 2025MMInference: Accelerating Pre-filling for Long-Context VLMs via Modality-Aware Permutation Sparse Attentioncited by 26
  5. 2024HashAttention: Semantic Sparsity for Faster Inferencecited by 23
  6. 2025Efficient Attention Mechanisms for Large Language Models: A Surveycited by 21
  7. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 18
  8. 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMscited by 18
  9. 2025Multipole Attention for Efficient Long Context Reasoningcited by 14
  10. 2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
  11. 2025RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inferencecited by 13
  12. 2025R3Mem: Bridging Memory Retention and Retrieval via Reversible Compressioncited by 12
  13. 2025Inference-time sparse attention with asymmetric indexingcited by 11
  14. 2025Beyond Exponential Decay: Rethinking Error Accumulation in Large Language Modelscited by 11
  15. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  16. 2025TailorKV: A Hybrid Framework for Long-Context Inference via Tailored KV Cache Optimizationcited by 10
  17. 2026Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Evictioncited by 10
  18. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Servingcited by 7
  19. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  20. 2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
  21. 2025Make It Efficient: Dynamic Sparse Attention for Autoregressive Image Generationcited by 7
  22. 2025VIBE: Vector Index Benchmark for Embeddingscited by 6
  23. 2025Draft-based Approximate Inference for LLMscited by 6
  24. 2025SCOPE: A Generative Approach for LLM Prompt Compressioncited by 6
  25. 2026FASA: Frequency-aware Sparse Attentioncited by 6
  26. 2025Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUscited by 5
  27. 2025LeanK: Learnable K Cache Channel Pruning for Efficient Decodingcited by 5
  28. 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inferencecited by 5
  29. 2025Sparse Attention across Multiple-context KV Cachecited by 5
  30. 2025SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Servingcited by 5
  31. 2025DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoningcited by 5
  32. 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decodingcited by 5
  33. 2026Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMscited by 5
  34. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inferencecited by 5
  35. 2024Bootstrap Your Own Context Lengthcited by 4
  36. 2025AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularitycited by 4
  37. 2025Retrospective Sparse Attention for Efficient Long-Context Generationcited by 4
  38. 2025MetaMuse: Algorithm Generation via Creative Ideationcited by 4

50 most cited of 114 All 114 (JSON)