mlsanityData to 14 Oct 2026

RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression

arXiv 2502.14051 ↗First posted 19 Feb 2025cs.CL cs.LGcited by 41

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Transformer-based Large Language Models rely critically on the KV cache to efficiently handle extended contexts during the decode phase. Yet, the size of the KV cache grows proportionally with the input length, burdening both memory bandwidth and capacity as decoding progresses. To address this challenge, we present RocketKV, a training-free KV cache compression strategy containing two consecutive stages. In the first stage, it performs coarse-grain permanent KV cache eviction on the input sequence tokens. In the second stage, it adopts a hybrid sparse attention method to conduct fine-grain top-k sparse attention, approximating the attention scores by leveraging both head and sequence dimensionality reductions. We show that RocketKV provides a compression ratio of up to 400×, end-to-end speedup of up to 3.7× as well as peak memory reduction of up to 32.6% in the decode phase on an NVIDIA A100 GPU compared to the full KV cache baseline, while achieving negligible accuracy loss on a variety of long-context tasks. We also propose a variant of RocketKV for multi-turn scenarios, which consistently outperforms other existing methods and achieves accuracy nearly on par with an oracle top-k attention scheme. The source code is available here: https://github.com/NVlabs/RocketKV.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 31 of its 41 references that are in the map. First posted on arXiv 19 Feb 2025.

Builds on

31 of 41 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving2 Jan
  2. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  3. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  4. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  5. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  6. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches1 Jul
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  8. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  9. 2024Block Transformer: Global-to-Local Language Modeling for Fast Inference4 Jun
  10. 2024Reducing Transformer Key-Value Cache Size with Cross-Layer Attention21 May
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  12. 2023SparQ Attention: Bandwidth-Efficient LLM Inference8 Dec
2 more
  1. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs3 Oct
  2. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2023Mistral 7Bcited by 3,164
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  8. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  9. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  10. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  11. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 196
  12. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
5 more
  1. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
  2. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  3. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  4. 2024SlimGPT: Layer-wise Structured Pruning for Large Language Modelscited by 44
  5. 2024The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Surveycited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 31.

41 papers.

Sort:

By area:

  1. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoningcited by 30
  2. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compressioncited by 21
  3. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 13
  4. 2025VideoNSA: Native Sparse Attention Scales Video Understandingcited by 11
  5. 2026FASA: Frequency-aware Sparse Attentioncited by 8
  6. 2026The Model Knows, the Decoder Finds: Future Value Guided Particle Power Samplingcited by 6
  7. 2026Slipstream: Trajectory-Grounded Compaction Validation for Long-Horizon Agentscited by 6
  8. 2025SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Modelscited by 5
  9. 2026SparDA: Sparse Decoupled Attention for Efficient Long-Context LLM Inferencecited by 5
  10. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 4
  11. 2025Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIMcited by 3
  12. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inferencecited by 3
29 more
  1. 2026Guess-Verify-Refine: Data-Aware Top-K for Sparse-Attention Decoding on Blackwell via Temporal Correlationcited by 3
  2. 2026Search Your Block Floating Point Scales!cited by 3
  3. 2025MatKV: Trading Compute for Flash Storage in LLM Inferencecited by 2
  4. 2026CompilerKV: Risk-Adaptive KV Compression via Offline Experience Compilationcited by 2
  5. 2026HARD-KV: Head-Adaptive Regularization for Decoding-time KV Compressioncited by 2
  6. 2026Towards Memory-Efficient Autoregressive Video Generation via Instance-Specific Parametric Absorptioncited by 2
  7. 2025Overcoming State Inertia: Minimally Invasive Temporal Alignment for Evolving Contextscited by 1
  8. 2025Unlocking the Address Book: Dissecting the Sparse Semantic Structure of LLM Key-Value Caches via Sparse Autoencoderscited by 1
  9. 2026Sublinear Time Quantum Algorithm for Attention Approximationcited by 1
  10. 2026CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inferencecited by 1
  11. 2026MAC-Attention: a Match-Amend-Complete Scheme for Fast and Accurate Attention Computationcited by 1
  12. 2026Transactional Attention: Semantic Sponsorship for KV-Cache Retentioncited by 1
  13. 2026Linear Scaling Video VLMs for Long Video Understandingcited by 1
  14. 2026Adaptive Filtering of the KV Cache: Diagnosing and Correcting Structural-Role Bias in LLM Inferencecited by 1
  15. 2026LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decodingcited by 1
  16. 2026Monte Carlo Estimation for KV Cache Evictioncited by 1
  17. 2025A Faster Generalized Two-Stage Approximate Top-Kcited by 0
  18. 2025KV Cache Recycling to Expand Usable Context Capacity in Low Parameter LLMscited by 0
  19. 2026PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selectioncited by 0
  20. 2026ETC: Extreme Token Compression via Task-aware Visual Information Distillation in VLMscited by 0
  21. 2026Codec-Gauge: Learning Compression-Friendly Gauges for Transformer KV Cachescited by 0
  22. 2026Prox: Training-Free FFN Activation Sparsity via Approximate Intermediate-Channel Salience in LLMscited by 0
  23. 2026S4R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Cachingcited by 0
  24. 2026DistillCache: KL-Guided Adaptive KV-Cache Eviction for Memory-Efficient LLM Inferencecited by 0
  25. 2026Parser Transitions Already Reveal Structural Responsibility: KV Persistence for Structured Generationcited by 0
  26. 2026What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservationcited by 0
  27. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inferencecited by 0
  28. 2026TierKV: Long-Context On-Device LLMs via Predictive Multi-Tier KV Cachingcited by 0
  29. 2026CORE: COverage CAlibration and Evicted-Mass REdistribution for KV Cachecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025KVCompose: Efficient Structured KV Cache Compression with Composite Tokens
  2. 2026SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
  3. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
  4. 2026PuzzleKV: Page-Wise Low-Rank Decomposition for KV Cache Compression
  5. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  6. 2026CompressKV: Semantic-Retrieval-Guided KV-Cache Compression for Resource-Efficient Long-Context LLM Inference
  7. 2026FreqDepthKV: Frequency-Guided Depth Sharing for Robust KV Cache Compression in Long-Context LLM Inference
  8. 2025HCAttention: Extreme KV Cache Compression via Heterogeneous Attention Computing for LLMs