mlsanityData to 7 Oct 2026

Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoning

arXiv 2503.04973 ↗First posted 6 Mar 2025cs.CL cs.AI cs.IRcited by 4

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Incorporating external knowledge in large language models (LLMs) enhances their utility across diverse applications, but existing methods have trade-offs. Retrieval-Augmented Generation (RAG) fetches evidence via similarity search, but key information may fall outside top ranked results. Long-context models can process multiple documents but are computationally expensive and limited by context window size. Inspired by students condensing study material for open-book exams, we propose task-aware key-value (KV) cache compression, which compresses external knowledge in a zero- or few-shot setup. This enables LLMs to reason efficiently over a compacted representation of all relevant information. Experiments show our approach outperforms both RAG and task-agnostic compression methods. On LongBench v2, it improves accuracy by up to 7 absolute points over RAG with a 30x compression rate, while reducing inference latency from 0.43s to 0.16s. A synthetic dataset highlights that RAG performs well when sparse evidence suffices, whereas task-aware compression is superior for broad knowledge tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 31 of its 50 references that are in the map. First posted on arXiv 6 Mar 2025.

Builds on

31 of 50 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning25 Oct
  2. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation17 Oct
  3. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul

Other references · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  3. 2024Qwen2.5 Technical Reportcited by 4,713
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  7. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  9. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,411
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2022A Survey on In-context Learningcited by 717
  12. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
16 more
  1. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  2. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  3. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  4. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  5. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  6. 2020Autoregressive Entity Retrievalcited by 333
  7. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  8. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 157
  9. 2024A Simple and Effective L2 Norm-Based Strategy for KV Cache Compressioncited by 84
  10. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  11. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 50
  12. 2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
  13. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  14. 2024Characterizing Prompt Compression Methods for Long Context Inferencecited by 18
  15. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Headcited by 12
  16. 2024Multi-Source Knowledge Pruning for Retrieval-Augmented Generation: A Benchmark and Empirical Studycited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025KVzip: Query-Agnostic KV Cache Compression with Context Reconstructioncited by 89
  2. 2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
  3. 2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
  4. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Cache-Augmented Generation (CAG) with Adaptive Contextual Compression for Scalable Knowledge Integration
  2. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression
  3. 2024ClusterKV: Manipulating LLM KV Cache in Semantic Space for Recallable Compression
  4. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks
  5. 2026Information-Aware KV Cache Compression for Long Reasoning
  6. 2025SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression
  7. 2025Less Is More: Elevating RAG via Performance-Driven Context Compression
  8. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoning