# Beyond Speedup -- Utilizing KV Cache for Sampling and Reasoning

arXiv: https://arxiv.org/abs/2601.20326 · HTML: https://mlsanity.com/paper/2601.20326/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 9 of its 28 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)
- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 9 of 9)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference](https://mlsanity.com/paper/2502.04420/) (2025)
- [When to Continue Thinking: Adaptive Thinking Mode Switching for Efficient Reasoning](https://mlsanity.com/paper/2505.15400/) (2025)
- [KV Cache Steering for Controlling Frozen LLMs](https://mlsanity.com/paper/2507.08799/) (2025)
- [PATS: Process-Level Adaptive Thinking Mode Switching](https://mlsanity.com/paper/2505.19250/) (2025)

## Built on by (most cited first, 3 of 3)

- [RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference](https://mlsanity.com/paper/2606.31519/) (2026)
- [Back from the Future: Key-Value Cache Management by Counter-Causal Surprise](https://mlsanity.com/paper/2607.27600/) (2026)
- [Can Computation from Earlier Problems Help LLMs Solve New Ones?](https://mlsanity.com/paper/2609.39394/) (2026)

## Abstract (arXiv)

KV caches, typically used only to speed up autoregressive decoding, encode contextual information that can be reused for downstream tasks at no extra cost. We propose treating the KV cache as a lightweight representation, eliminating the need to recompute or store full hidden states. Despite being weaker than dedicated embeddings, KV-derived representations are shown to be sufficient for two key applications: (i) Chain-of-Embedding, where they achieve competitive or superior performance on Llama-3.1-8B-Instruct and Qwen2-7B-Instruct; and (ii) Fast/Slow Thinking Switching, where they enable adaptive reasoning on Qwen3-8B and DeepSeek-R1-Distil-Qwen-14B, reducing token generation by up to 5.7× with minimal accuracy loss. Our findings establish KV caches as a free, effective substrate for sampling and reasoning, opening new directions for representation reuse in LLM inference. Code: https://github.com/cmd2001/ICLR2026KV-Embedding.