Data to 7 Oct 2026

Not All Heads Matter: A Head-Level KV Cache Compression Method with Integrated Retrieval and Reasoning

arXiv 2410.19258 ↗First posted 25 Oct 2024cs.CL cs.AIcited by 96

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more

Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Key-Value (KV) caching is a common technique to enhance the computational efficiency of Large Language Models (LLMs), but its memory overhead grows rapidly with input length. Prior work has shown that not all tokens are equally important for text generation, proposing layer-level KV cache compression to selectively retain key information. Recognizing the distinct roles of attention heads in generation, we propose HeadKV, a head-level KV cache compression method, and HeadKV-R2, which leverages a novel contextual reasoning ability estimation for compression. Our approach operates at the level of individual heads, estimating their importance for contextual QA tasks that require both retrieval and reasoning capabilities. Extensive experiments across diverse benchmarks (LongBench, LooGLE), model architectures (e.g., Llama-3-8B-Instruct, Mistral-7B-Instruct), and long-context abilities tests demonstrate that our head-level KV cache compression significantly outperforms strong baselines, particularly in low-resource settings (KV size = 64 & 128). Notably, our method retains just 1.5% of the KV cache while achieving 97% of the performance of the full KV cache on the contextual question answering benchmark. Codes are available at https://github.com/FYYFU/HeadKV

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 36 of its 40 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

36 of 40 references.

References · 36

2017Attention Is All You Needcited by 59,932
2023GPT-4 Technical Reportcited by 18,132
2024The Llama 3 Herd of Modelscited by 14,192
2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
2024Qwen2 Technical Reportcited by 2,076
2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844

24 more

2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
2024A Survey on Efficient Inference for Large Language Modelscited by 197
2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 190
2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
2022A Fast Post-Training Pruning Framework for Transformerscited by 161
2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
2024RazorAttention: Efficient KV Cache Compression Through Retrieval Headscited by 99
2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
2024ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partitioncited by 81
2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
2024Attention Heads of Large Language Models: A Surveycited by 65
2019Extractive Summarization of Long Documents by Combining Global and Local Contextcited by 59
2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
2021Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modelingcited by 14
2024StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Lossescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 46; 2026: 47.

96 papers.

Sort:

By area:

2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
2025InfiniPot-V: Memory-Constrained KV Cache Compression for Streaming Video Understandingcited by 54
2025StreamMem: Query-Agnostic KV Cache Memory for Streaming Video Understandingcited by 53
2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Modelscited by 52
2025CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspectivecited by 28
2025StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compressioncited by 24
2025Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compressioncited by 22
2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
2025Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMscited by 20
2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoningcited by 17
2025Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoningcited by 16

38 more

2025ReCalKV: Low-Rank KV Cache Compression via Head Reordering and Offline Calibrationcited by 16
2025Head-Aware KV Cache Compression for Efficient Visual Autoregressive Modelingcited by 15
2025Taming the Fragility of KV Cache Eviction in LLM Inferencecited by 14
2025FastKV: Decoupling of Context Reduction and KV Cache Compression for Prefill-Decoding Accelerationcited by 13
2025Mixing Importance with Diversity: Joint Optimization for KV Cache Compression in Large Vision-Language Modelscited by 13
2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compressioncited by 10
2025Streaming Attention Approximation via Discrepancy Theorycited by 9
2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
2025OBCache: Optimal Brain KV Cache Pruning for Efficient Long-Context LLM Inferencecited by 9
2025KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inferencecited by 8
2025UniGist: Towards General and Hardware-aligned Sequence-level Long Context Compressioncited by 8
2025Mustafar: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inferencecited by 7
2025G-KV: Decoding-Time KV Cache Eviction with Global Attentioncited by 7
2025Adaptive Computation Pruning for the Forgetting Transformercited by 6
2025Draft-based Approximate Inference for LLMscited by 6
2025SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruningcited by 6
2026MuKV: Multi-Grained KV Cache Compression for Long Streaming Video Question-Answeringcited by 6
2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttentioncited by 6
2025AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selectioncited by 5
2026From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generationcited by 5
2026When Less Latent Leads to Better Relay: Information-Preserving Compression for Latent Multi-Agent LLM Collaborationcited by 5
2024EMS: Adaptive Evict-then-Merge Strategy for Head-wise KV Cache Compression Based on Global-Local Importancecited by 4
2025Beyond RAG: Task-Aware KV Cache Compression for Comprehensive Knowledge Reasoningcited by 4
2025Harnessing the Computation Redundancy in ViTs to Boost Adversarial Transferabilitycited by 4
2025AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularitycited by 4
2026Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Evictioncited by 4
2026Nearly Optimal Attention Coresetscited by 4
2026LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Evictioncited by 4
2026GRKV: Global Regression for Training-Free KV Cache Compression in Long-Context LLMscited by 4
2025ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Modelscited by 3
2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 3
2026Adaptive Layer Selection for Layer-Wise Token Pruning in LLM Inferencecited by 3
2026From Static Inference to Dynamic Interaction: A Survey of Streaming Large Language Modelscited by 3
2025Activation-aware Probe-Query: Effective Key-Value Retrieval for Long-Context LLMs Inferencecited by 2
2025ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Modelscited by 2
2025REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compressioncited by 2
2025AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generationcited by 2

50 most cited of 96 All 96 (JSON)

Similar, not cited

Close in wording, no citation either way