mlsanityData to 14 Oct 2026

RetroInfer: A Vector Storage Engine for Scalable Long-Context LLM Inference

arXiv 2505.02922 ↗First posted 5 May 2025cs.LGcited by 13

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Recent large language models (LLMs) are rapidly extending their context windows, yet inference throughput lags due to increasing GPU memory and bandwidth demands. This is because the key-value (KV) cache, an intermediate structure storing token representations, grows linearly with context length and requires an iterative linear scan for attention computation. A promising direction to accelerate long-context inference is to exploit attention's inherent sparsity by offloading the KV cache to CPU memory and retrieving only a small subset of tokens important to the current generation step. However, prior sparse attention approaches struggle to balance accuracy and retrieval cost due to varying sparsity patterns and inefficient GPU-CPU memory management. We present RetroInfer, a vector storage engine that realizes a sparsity-based KV cache for long-context inference. RetroInfer introduces an Attention-aWare VEctor index (wave index), which fundamentally improves the tradeoff between attention accuracy and retrieval cost through tripartite attention approximation, accuracy-bound attention estimation, and segmented clustering. We also design the wave buffer, a GPU-CPU buffer manager that assigns computation and manages data across heterogeneous hardware. We evaluate RetroInfer across a range of models and workloads, demonstrating up to 4.4X decoding throughput over full attention at 120K context and up to 12.2X over sparse attention baselines at 1 million tokens -- all while preserving full-attention-level accuracy.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 63 of its 115 references that are in the map. First posted on arXiv 5 May 2025.

Builds on

63 of 115 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs17 Feb
  2. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  3. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  4. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection5 Nov
  5. 2024Fast State Restoration in LLM Serving with HCache7 Oct
  6. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  7. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling4 Jun
  9. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  10. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention23 Mar
  11. 2023Stateful Large Language Model Serving with Pensieve9 Dec
  12. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
1 more
  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 50

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  10. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  11. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 688
  12. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
38 more
  1. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  3. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  5. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  6. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  7. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  8. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  9. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 217
  10. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  11. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197
  12. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  13. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 177
  14. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  15. 2023CodePlan: Repository-level Coding using LLMs and Planningcited by 140
  16. 2025XAttention: Block Sparse Attention with Antidiagonal Scoringcited by 137
  17. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  18. 2024RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrievalcited by 116
  19. 2024LoongServe: Efficiently Serving Long-Context Large Language Models with Elastic Sequence Parallelismcited by 112
  20. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  21. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  22. 2017Fast Approximate Nearest Neighbor Search With The Navigating Spreading-out Graphcited by 104
  23. 2024PowerInfer-2: Fast Large Language Model Inference on a Smartphonecited by 103
  24. 2024PQCache: Product Quantization-based KVCache for Long Context LLM Inferencecited by 95
  25. 2023Fairness in Serving Large Language Modelscited by 78
  26. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  27. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
  28. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  29. 2025Understanding Aha Moments: from External Observations to Internal Mechanismscited by 50
  30. 2022Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutionscited by 38
  31. 2024ACORN: Performant and Predicate-Agnostic Search Over Vector Embeddings and Structured Datacited by 32
  32. 2022Merlin HugeCTR: GPU-accelerated Recommender System Training and Inferencecited by 20
  33. 2023High-Throughput Vector Similarity Search in Knowledge Graphscited by 19
  34. 2024RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Searchcited by 12
  35. 2024DET-LSH: A Locality-Sensitive Hashing Scheme with Dynamic Encoding Tree for Approximate Nearest Neighbor Searchcited by 6
  36. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Servingcited by 6
  37. 2025AlayaDB: The Data Foundation for Efficient and Effective Long-context LLM Inferencecited by 3
  38. 2025TigerVector: Supporting Vector Search in Graph Databases for Advanced RAGscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  2. 2026Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attentioncited by 7
  3. 2026KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inferencecited by 5
  4. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
  5. 2026Adaptive KV Cache Reuse for Fast Long-Context LLM Servingcited by 3
  6. 2026Multi-Segment Attention: Enabling Efficient KV-Cache Management for Faster Large Language Model Servingcited by 3
  7. 2026SmartGen: Seamless Disaggregated LLM Inference with Selective KV Cache Transfercited by 2
  8. 2026H2MT: Semantic Hierarchy-Aware Hierarchical Memory Transformercited by 1
  9. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0
  10. 2026Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Cachescited by 0
  11. 2026PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attentioncited by 0
  12. 2026SparseEngine: Sparse-First Inference Enginecited by 0
1 more
  1. 2026MOIRA: Mass-Oriented Indexing with Ragged Attention for Long-Context Decodingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking the Boundaries of Long-Context LLM Inference: Adaptive KV Management on a Single Commodity GPU
  2. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts
  3. 2026ARKV: Adaptive and Resource-Efficient KV Cache Management under Limited Memory Budget for Long-Context Inference in LLMs
  4. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads
  5. 2025HGCA: Hybrid GPU-CPU Attention for Long Context LLM Inference
  6. 2025MoSKA: Mixture of Shared KV Attention for Efficient Long-Sequence LLM Inference
  7. 2025RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression
  8. 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading