mlsanityData to 14 Oct 2026

Inference-Time Hyper-Scaling with KV Cache Compression

arXiv 2506.05345 ↗First posted 5 Jun 2025cs.LG cs.CLcited by 27

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Inference-time scaling trades efficiency for increased reasoning accuracy by generating longer or more parallel sequences. However, in Transformer LLMs, generation cost is bottlenecked by the size of the key-value (KV) cache, rather than the number of generated tokens. Hence, we explore inference-time hyper-scaling: by compressing the KV cache, we can generate more tokens within the same compute budget and further improve the accuracy of scaled inference. The success of this approach, however, hinges on the ability of compression methods to preserve accuracy even at high compression ratios. To make hyper-scaling practical, we introduce Dynamic Memory Sparsification (DMS), a novel method for sparsifying KV caches that only requires 1K training steps to achieve 8× compression, while maintaining better accuracy than training-free sparse attention. Instead of prematurely discarding cached tokens, DMS delays token eviction, implicitly merging representations and preserving critical information. We demonstrate the effectiveness of inference-time hyper-scaling with DMS on multiple families of LLMs, showing that it boosts accuracy for comparable inference latency and memory load. For instance, we enhance Qwen-R1 32B by 12.0 points on AIME 24, 8.6 on GPQA, and 9.7 on LiveCodeBench on average for an equivalent number of memory reads.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 47 of its 60 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

47 of 60 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals18 Apr
  2. 2024Improving LLM Reasoning through Scaling Inference Computation with Collaborative Verification5 Oct
  3. 2024Eigen Attention: Attention in Low-Rank Space for KV Cache Compression10 Aug
  4. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  5. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training29 Sep
  6. 2023Self-Refine: Iterative Refinement with Self-Feedback30 Mar

Other references · 41

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2023Let's Verify Step by Stepcited by 3,789
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
29 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  8. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  9. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  11. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  13. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  14. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  15. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  16. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  17. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  18. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
  19. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  20. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  21. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  22. 2024Flex Attention: A Programming Model for Generating Optimized Attention Kernelscited by 185
  23. 2024Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inferencecited by 116
  24. 2024Transformers need glasses! Information over-squashing in language taskscited by 91
  25. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 91
  26. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  27. 2024LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategycited by 51
  28. 2025The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMscited by 29
  29. 2024FlashMask: Efficient and Rich Mask Extension of FlashAttentioncited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 18.

27 papers.

Sort:

By area:

  1. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 74
  2. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Evictioncited by 16
  3. 2026KVzap: Fast, Adaptive, and Faithful KV Cache Pruningcited by 15
  4. 2025The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoningcited by 13
  5. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 13
  6. 2025BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholdingcited by 13
  7. 2025Fast and Expressive Multi-Byte Prediction with Probabilistic Circuitscited by 9
  8. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 9
  9. 2026Position: Modular Memory is the Key to Continual Learning Agentscited by 8
  10. 2026Accordion-Thinking: Self-Regulated Step Summaries for Efficient and Readable LLM Reasoningcited by 6
  11. 2026Cartridges at Scale: Training Modular KV Caches over Large Document Collectionscited by 6
  12. 2025KV Admission: Learning What to Write for Efficient Long-Context LLM Inferencecited by 4
15 more
  1. 2026DashAttention: Differentiable and Adaptive Sparse Hierarchical Attentioncited by 4
  2. 2025Alleviating Forgetfulness of Linear Attention by Hybrid Sparse Attention and Contextualized Learnable Token Evictioncited by 3
  3. 2025AsyncSpade: Efficient Test-Time Scaling with Asynchronous Sparse Decodingcited by 2
  4. 2026Neural Garbage Collection: Learning to Forget while Learning to Reasoncited by 2
  5. 2026Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utilitycited by 2
  6. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compressioncited by 2
  7. 2026Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operatorscited by 1
  8. 2026Linear Scaling Video VLMs for Long Video Understandingcited by 1
  9. 2026KVDiagnosis: A Diagnostic Benchmark for KV-Cache Compression in Long-Context Language Modelscited by 1
  10. 2026EchoPress: Query-Agnostic KV Cache Pruning via Virtual Context Reconstructioncited by 1
  11. 2026Behavior-Preserving KV Cache Compressioncited by 1
  12. 2026KVSlimmer: Theoretical Insights and Practical Optimizations for Asymmetric KV Mergingcited by 0
  13. 2026Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inferencecited by 0
  14. 2026Diffusion-Guided Uncertainty-Aware Delayed Policy Optimizationcited by 0
  15. 2026Learning how to Forget: Fine-tuning for Long-Context Sparse Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  2. 2025Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving
  3. 2026MetaKV: Adaptive KV Cache Compression for Constrained LLM Inference
  4. 2024RefreshKV: Updating Small KV Cache During Long-form Generation
  5. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inference
  6. 2025KV Cache Compression for Inference Efficiency in LLMs: A Review
  7. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Models
  8. 2026C2KV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference