mlsanityData to 14 Oct 2026

FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference

arXiv 2505.13109 ↗First posted 19 May 2025cs.LG cs.AI cs.CLcited by 14

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Large language models (LLMs) are widely deployed with rapidly expanding context windows to support increasingly demanding applications. However, long contexts pose significant deployment challenges, primarily due to the KV cache whose size grows proportionally with context length. While KV cache compression methods have been proposed to address this issue, KV dropping methods incur considerable accuracy loss, and KV retrieval methods suffer from significant efficiency bottlenecks. We propose FreeKV, a training-free algorithm-system co-optimization framework to enhance KV retrieval efficiency while preserving accuracy. On the algorithm side, FreeKV introduces speculative retrieval to shift the KV selection and recall processes out of the critical path, combined with fine-grained correction to ensure accuracy. On the system side, FreeKV employs hybrid KV layouts across CPU and GPU memory to eliminate fragmented data transfers, and leverages double-buffered streamed recall to further improve efficiency, enabling effective overlap with computation, full latency hiding, and practical speedups from speculative recall. Experiments demonstrate that FreeKV achieves near-lossless accuracy across various scenarios and models, delivering up to a 13× speedup compared to SOTA KV retrieval methods. Code is available at https://github.com/sjtu-zhao-lab/FreeKV.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 32 of its 49 references that are in the map. First posted on arXiv 19 May 2025.

Builds on

32 of 49 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Serving31 Mar
  2. 2025TokenButler: Token Importance is Predictable10 Mar
  3. 2025Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving1 Mar
  4. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methods13 Dec
  5. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression4 Dec
  6. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference28 Oct
  7. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads14 Oct
  8. 2024RazorAttention: Efficient KV Cache Compression Through Retrieval Heads22 Jul
  9. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference16 Jul
  10. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management28 Jun
  11. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference16 Jun
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 20

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  8. 2024A Survey on Large Language Models for Code Generationcited by 658
  9. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  10. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  11. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  12. 2024SeerAttention: Learning Intrinsic Sparse Attention in Your LLMscited by 124
8 more
  1. 2024MagicPIG: LSH Sampling for Efficient LLM Generationcited by 111
  2. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  3. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
  4. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 57
  5. 2025Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruningcited by 49
  6. 2024LongGenBench: Benchmarking Long-Form Generation in Long Context LLMscited by 47
  7. 2024Challenges in Deploying Long-Context Transformers: A Theoretical Peak Performance Analysiscited by 44
  8. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compressioncited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2026MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answeringcited by 6
  2. 2025SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Modelscited by 5
  3. 2026KEEP: A KV-Cache-Centric Memory Management System for Efficient Embodied Planningcited by 5
  4. 2025LouisKV: Efficient KV Cache Retrieval for Long Input-Output Sequencescited by 4
  5. 2026ParisKV: Fast and Drift-Robust KV-Cache Retrieval for Long-Context LLMscited by 2
  6. 2026AB-Sparse: Sparse Attention with Adaptive Block Size for Accurate and Efficient Long-Context Inferencecited by 2
  7. 2026Epiphany-Aware KV Cache Eviction Without the Attention Matrixcited by 2
  8. 2026TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inferencecited by 1
  9. 2026Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cachecited by 1
  10. 2026Near-Oracle KV Selection via Pre-hoc Sparsity for Long-Context Inferencecited by 0
  11. 2026PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selectioncited by 0
  12. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0
2 more
  1. 2026CommunityKV: Efficient Long-Context Decoding via Graph Partitioningcited by 0
  2. 2026OVAL: Output-Aware Local Page Bases for KV Cache Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
  3. 2025BaKlaVa -- Budgeted Allocation of KV cache for Long-context Inference
  4. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  5. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
  6. 2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
  7. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression
  8. 2025ReFreeKV: Towards Threshold-Free KV Cache Compression