mlsanityData to 7 Oct 2026

Spend Bytes on Breadth: Precision-Count Trade-offs for Decode-Time KV Compression in Long Chain-of-Thought Reasoning

arXiv 2610.05685 ↗First posted 5 Oct 2026cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Reasoning models write most of their KV cache while decoding long chains of thought (CoT), so the cache has to be compressed online under a fixed memory budget. Decode-time methods mostly decide which tokens to evict. We ask how a fixed byte budget should be split between the number of cached tokens and their precision. BreadthKV spends the bytes on more tokens at low precision, combining quantization with eviction, and picks the bit-width for each model and budget with a 60-problem end-to-end calibration, since offline attention error does not predict it reliably. On three reasoning models and four math and science benchmarks, it scores above eviction alone in 17 of 18 settings and produces shorter outputs. Much of what eviction loses comes from derailed runs, which keep reasoning until the length cap without reaching an answer. On Qwen3-8B at our tightest budget, eviction sends 91% of AIME samples to the cap and BreadthKV 40%. Under the same protocol, BreadthKV is statistically indistinguishable from a joint rate-distortion allocator (RDKV) that uses 27% more KV memory-time, and it outperforms our re-implementation of ThinKV.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 52 of its 61 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

52 of 61 references.

Closest ancestors · 22

Same direction, newest first

  1. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning3 Sep
  2. 2026KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving16 Aug
  3. 2026Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning5 Aug
  4. 2026Epiphany-Aware KV Cache Eviction Without the Attention Matrix25 Jun
  5. 2026Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models24 Jun
  6. 2026ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for Reasoning Models9 Jun
  7. 2026Value-Aware Stochastic KV Cache Eviction for Reasoning Models2 Jun
  8. 2026KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks2 Jun
  9. 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compression6 Apr
  10. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoning22 Dec
  11. 2025Wait, Wait, Wait... Why Do Reasoning Models Loop?15 Dec
  12. 2025G-KV: Decoding-Time KV Cache Eviction with Global Attention29 Nov
10 more
  1. 2025Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving8 Nov
  2. 2025Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models13 Oct
  3. 2025Which Heads Matter for Reasoning? RL-Guided KV Cache Compression9 Oct
  4. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models1 Oct
  5. 2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning19 Jun
  6. 2025SeerAttention-R: Sparse Attention Adaptation for Long Reasoning10 Jun
  7. 2025R-KV: Redundancy-aware KV Cache Compression for Reasoning Models30 May
  8. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs24 May
  9. 2025RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning16 Feb
  10. 2025Semantic Integrity Matters: Benchmarking and Preserving High-Density Reasoning in KV Cache Compression4 Feb

Other references · 30

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  8. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  9. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  10. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  11. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  12. 2024PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inferencecited by 148
18 more
  1. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  2. 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
  3. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  4. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  5. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  6. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  7. 2025xKV: Cross-Layer KV-Cache Compression via Aligned Singular Vector Extractioncited by 13
  8. 2025KV Cache Transform Coding for Compact Storage in LLM Inferencecited by 11
  9. 2026STAR-KV: Low-Rank KV Cache Compression via Soft Thresholding for Adaptive Rank Controlcited by 11
  10. 2024More Tokens, Lower Precision: Towards the Optimal Token-Precision Trade-off in KV Cache Compressioncited by 10
  11. 2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boostcited by 8
  12. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  13. 2025Value-Guided KV Compression for LLMs via Approximated CUR Decompositioncited by 6
  14. 2026RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cachecited by 4
  15. 2026TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Taskscited by 3
  16. 2026SPHERICAL KV: Angle-Domain Attention and Rate-Distortion Retention for Efficient Long-Context Inferencecited by 2
  17. 2026Does Accuracy Equal Evidence? Reasoning Faithfulness under KV Cache Compressioncited by 2
  18. 2026QEvict: Recoverable Quantized KV Eviction for Attention-Drift-Robust Long-Context Decodingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
  2. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
  3. 2025R1-Compress: Long Chain-of-Thought Compression via Chunk Compression and Search
  4. 2026BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
  5. 2026Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoning
  6. 2025When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models
  7. 2026Quantized Reasoning Models Think They Need to Think Longer, but They Do Not
  8. 2025TokenSkip: Controllable Chain-of-Thought Compression in LLMs