mlsanityData to 7 Oct 2026

Thought-Aware KV Cache Compaction for Reasoning via Adaptive Attention Matching

arXiv 2608.12331 ↗First posted 1 Jun 2026cs.CL cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Reasoning language models generate lengthy chain-of-thought (CoT) sequences whose key-value (KV) cache grows linearly and becomes a memory bottleneck during decoding. Existing compaction methods treat reasoning trajectories as flat token sequences and apply uniform compression, ignoring the hierarchical structure of CoT reasoning where different steps vary drastically in importance. We propose Thought-Aware Attention Matching (TAM), which exploits this structure through three mechanisms: (i)~thought segmentation that decomposes the trajectory into reasoning blocks, (ii)~adaptive budget allocation that assigns compression budget based on each segment's importance and size, and (iii)~pivotal token protection that preserves high-attention reasoning anchors. We prove that the allocation rule is optimal under a convex error model and that cumulative error under sequential compaction remains bounded. Experiments on AIME 2024 and MATH-500 with Qwen3-4B show that TAM improves accuracy over uniform compaction at the same memory footprint, with periodic compaction bounding peak memory to 3.1--3.2,GB (a 65% reduction) while maintaining competitive accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 9 of its 19 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

9 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoning20 May

Other references · 8

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  6. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  7. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  8. 2026Fast KV Compaction via Attention Matchingcited by 28

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models
  2. 2026Crystal-KV: Efficient KV Cache Management for Chain-of-Thought LLMs via Answer-First Principle
  3. 2026Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning
  4. 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
  5. 2026BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inference
  6. 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoning
  7. 2025Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Serving
  8. 2025REAL: REtrieval-reAsoning and Logic-constructed Attention Behaviors for Long-Context KV Cache Compression