# Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts

arXiv: https://arxiv.org/abs/2509.21743 · HTML: https://mlsanity.com/paper/2509.21743/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 37 of its 47 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 30 of 37)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [The Lessons of Developing Process Reward Models in Mathematical Reasoning](https://mlsanity.com/paper/2501.07301/) (2025)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [Token-Budget-Aware LLM Reasoning](https://mlsanity.com/paper/2412.18547/) (2024)
- [Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning](https://mlsanity.com/paper/2405.00451/) (2024)
- [Thinkless: LLM Learns When to Think](https://mlsanity.com/paper/2505.13379/) (2025)
- [Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents](https://mlsanity.com/paper/2310.19923/) (2023)
- [Efficient Reasoning Models: A Survey](https://mlsanity.com/paper/2504.10903/) (2025)
- [RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation](https://mlsanity.com/paper/2403.05313/) (2024)
- [Effectively Controlling Reasoning Models through Thinking Intervention](https://mlsanity.com/paper/2503.24370/) (2025)
- [ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates](https://mlsanity.com/paper/2502.06772/) (2025)
- [MathBERT: A Pre-Trained Model for Mathematical Formula Understanding](https://mlsanity.com/paper/2105.00377/) (2021)
- [Enhancing LLM Reasoning with Reward-guided Tree Search](https://mlsanity.com/paper/2411.11694/) (2024)
- [Learning Adaptive Parallel Reasoning with Language Models](https://mlsanity.com/paper/2504.15466/) (2025)
- [When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs](https://mlsanity.com/paper/2505.11423/) (2025)
- [SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction](https://mlsanity.com/paper/2410.09008/) (2024)
- [Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.24726/) (2025)
- [AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time](https://mlsanity.com/paper/2505.24863/) (2025)
- [FlashThink: An Early Exit Method For Efficient Reasoning](https://mlsanity.com/paper/2505.13949/) (2025)
- [Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models](https://mlsanity.com/paper/2505.14810/) (2025)
- [Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning](https://mlsanity.com/paper/2106.03921/) (2021)
- [Value-Guided Search for Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2505.17373/) (2025)

## Built on by (most cited first, 12 of 12)

- [UMEM: Unified Memory Extraction and Management Framework for Generalizable Memory](https://mlsanity.com/paper/2602.10652/) (2026)
- [Procedural Knowledge at Scale Improves Reasoning](https://mlsanity.com/paper/2604.01348/) (2026)
- [Measuring Weak-to-Strong Legibility of Reasoning Models](https://mlsanity.com/paper/2603.20508/) (2026)
- [Multi-Agent Transactive Memory](https://mlsanity.com/paper/2606.19911/) (2026)
- [RAG over Thinking Traces Can Improve Reasoning Tasks](https://mlsanity.com/paper/2605.03344/) (2026)
- [Faster Synchronous On-Policy RL via Straggler-Aware Group Sizing](https://mlsanity.com/paper/2606.02218/) (2026)
- [Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents](https://mlsanity.com/paper/2608.20274/) (2026)
- [Big Reasoning with Small Models: Instruction Retrieval at Inference Time](https://mlsanity.com/paper/2510.13935/) (2025)
- [TIDE: Proactive Multi-Problem Discovery via Template-Guided Iteration](https://mlsanity.com/paper/2606.04743/) (2026)
- [MILES: Modular Instruction Memory with Learnable Selection for Self-Improving LLM Reasoning](https://mlsanity.com/paper/2607.06974/) (2026)
- [Structural Rationale Distillation via Reasoning Space Compression](https://mlsanity.com/paper/2605.07139/) (2026)
- [Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoning](https://mlsanity.com/paper/2608.21265/) (2026)

## Abstract (arXiv)

Large reasoning models improve accuracy by producing long reasoning traces, but this inflates latency and cost, motivating inference-time efficiency. We propose Retrieval-of-Thought (RoT), which reuses prior reasoning as composable ``thought" steps to guide new problems. RoT organizes steps into a thought graph with sequential and semantic edges to enable fast retrieval and flexible recombination. At inference, RoT retrieves query-relevant nodes and applies reward-guided traversal to assemble a problem-specific template that guides generation. This dynamic template reuse reduces redundant exploration and, therefore, reduces output tokens while preserving accuracy. We evaluate RoT on reasoning benchmarks with multiple models, measuring accuracy, token usage, latency, and memory overhead. Findings show small prompt growth but substantial efficiency gains, with RoT reducing output tokens by up to 40%, inference latency by 82%, and cost by 59% while maintaining accuracy. RoT establishes a scalable paradigm for efficient LRM reasoning via dynamic template construction through retrieval.