Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning
On the map: LLM reasoning → Test-time compute and efficient thinking
Abstract (arXiv)
Reasoning LLMs produce thousands of chain-of-thought tokens whose KV cache must reside in scarce GPU HBM. The dominant response -- permanently evicting low-importance tokens -- is catastrophic for reasoning: accuracy collapses to 0-2.5% when half the cache is removed. We ask a different question: must every token live in HBM, or can some live elsewhere? We introduce a semantics-aware memory hierarchy that sorts tokens into four tiers -- HBM, DDR, compressed, and evicted -- using cumulative attention scoring. Low-importance tokens are moved to CPU memory rather than destroyed; before each attention step they are prefetched back at full precision, contributing exactly the same terms as if they had never left the GPU. We formalize this as zero-approximation-error offloading and derive our central finding: accuracy depends solely on how many tokens are permanently discarded (the eviction ratio), not on how many remain in HBM. A controlled 3x3 grid over HBM and eviction ratios confirms this across three model scales (7B-32B) and four benchmarks. With only 3% eviction, the hierarchy retains 91% of full-cache accuracy on GSM8K and 71% on MATH-500 (n=200); at 14B scale it matches the uncompressed baseline (90% vs. 86%) while halving HBM occupancy. A head-to-head reproduction of R-KV -- the current SOTA eviction method -- on our setup achieves only 0-32% at comparable budgets. A system prototype with real GPU-CPU data movement shows that the price of this preservation is modest -- 5-7% transfer overhead -- and scaling analysis projects 2-48 GB HBM savings at production batch sizes.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 9 of its 22 references that are in the map. First posted on arXiv 10 May 2026.
Builds on
9 of 22 references.
Closest ancestors · 2
Same direction, newest first
- 2026TriAttention: Efficient Long Reasoning with Trigonometric KV Compression6 Apr
- 2025Hold Onto That Thought: Assessing KV Cache Compression On Reasoning12 Dec
Other references · 7
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
- 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
- 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
- 2025HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloadingcited by 9
- 2025ELMo-Tune-V2: LLM-Assisted Full-Cycle Auto-Tuning to Optimize LSM-Based Key-Value Storescited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference
- 2026ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning
- 2025Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs
- 2026KARA: Efficient Reasoning LLM Serving via Sliding-Window KV Cache Compression
- 2026Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention
- 2026Crystal-KV: Efficient KV Cache Management for Chain-of-Thought LLMs via Answer-First Principle
- 2025LazyEviction: Lagged KV Eviction with Attention Pattern Observation for Efficient Long Reasoning
- 2026MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration