mlsanityData to 7 Oct 2026

Retrieval-of-Thought: Efficient Reasoning via Reusing Thoughts

arXiv 2509.21743 ↗First posted 26 Sep 2025cs.AI cs.LGcited by 12

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Large reasoning models improve accuracy by producing long reasoning traces, but this inflates latency and cost, motivating inference-time efficiency. We propose Retrieval-of-Thought (RoT), which reuses prior reasoning as composable ``thought" steps to guide new problems. RoT organizes steps into a thought graph with sequential and semantic edges to enable fast retrieval and flexible recombination. At inference, RoT retrieves query-relevant nodes and applies reward-guided traversal to assemble a problem-specific template that guides generation. This dynamic template reuse reduces redundant exploration and, therefore, reduces output tokens while preserving accuracy. We evaluate RoT on reasoning benchmarks with multiple models, measuring accuracy, token usage, latency, and memory overhead. Findings show small prompt growth but substantial efficiency gains, with RoT reducing output tokens by up to 40%, inference latency by 82%, and cost by 59% while maintaining accuracy. RoT establishes a scalable paradigm for efficient LRM reasoning via dynamic template construction through retrieval.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 47 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

37 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Hybrid Deep Searcher: Scalable Parallel and Sequential Search Reasoning26 Aug
  2. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation8 Mar

Other references · 35

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  7. 2024OpenAI o1 System Cardcited by 1,220
  8. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  9. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  10. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  11. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  12. 2024Token-Budget-Aware LLM Reasoningcited by 242
23 more
  1. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  2. 2025Thinkless: LLM Learns When to Thinkcited by 107
  3. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 103
  4. 2025Efficient Reasoning Models: A Surveycited by 93
  5. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  6. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 62
  7. 2021MathBERT: A Pre-Trained Model for Mathematical Formula Understandingcited by 58
  8. 2024Enhancing LLM Reasoning with Reward-guided Tree Searchcited by 58
  9. 2025Learning Adaptive Parallel Reasoning with Language Modelscited by 58
  10. 2025When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMscited by 51
  11. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 32
  12. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 32
  13. 2025FlashThink: An Early Exit Method For Efficient Reasoningcited by 28
  14. 2025Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Modelscited by 28
  15. 2025AlphaOne: Reasoning Models Thinking Slow and Fast at Test Timecited by 27
  16. 2021Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoningcited by 23
  17. 2025Value-Guided Search for Efficient Chain-of-Thought Reasoningcited by 12
  18. 2025ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMscited by 11
  19. 2025Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNetscited by 10
  20. 2025Adaptive Termination for Multi-round Parallel Reasoning: An Universal Semantic Entropy-Guided Frameworkcited by 10
  21. 2024Chain of Evidences and Evidence to Generate: Prompting for Context Grounded and Retrieval Augmented Reasoningcited by 7
  22. 2025Predictive Scaling Laws for Efficient GRPO Training of Large Reasoning Modelscited by 6
  23. 2025Accelerating LLM Reasoning via Early Rejection with Partial Reward Modelingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 2026UMEM: Unified Memory Extraction and Management Framework for Generalizable Memorycited by 7
  2. 2026Procedural Knowledge at Scale Improves Reasoningcited by 7
  3. 2026Measuring Weak-to-Strong Legibility of Reasoning Modelscited by 5
  4. 2026RAG over Thinking Traces Can Improve Reasoning Taskscited by 2
  5. 2026Faster Synchronous On-Policy RL via Straggler-Aware Group Sizingcited by 2
  6. 2025Big Reasoning with Small Models: Instruction Retrieval at Inference Timecited by 1
  7. 2026Multi-Agent Transactive Memorycited by 1
  8. 2026Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agentscited by 1
  9. 2026Structural Rationale Distillation via Reasoning Space Compressioncited by 0
  10. 2026TIDE: Proactive Multi-Problem Discovery via Template-Guided Iterationcited by 0
  11. 2026MILES: Modular Instruction Memory with Learnable Selection for Self-Improving LLM Reasoningcited by 0
  12. 2026Memory Augmentation Unlocks Efficient Chain-of-Thought Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ThinkRetrieve: Retrieval-Augmented Reasoning Traces for Test-Time Scaling
  2. 2026When to Retrieve During Reasoning: Adaptive Retrieval for Large Reasoning Models
  3. 2026Compute Allocation for Reasoning-Intensive Retrieval Agents
  4. 2026Thinking with Reasoning Skills: Fewer Tokens, More Accuracy
  5. 2026Adaptive Retrieval helps Reasoning in LLMs -- but mostly if it's not used
  6. 2025When Thoughts Meet Facts: Reusable Reasoning for Long-Context LMs
  7. 2025Adaptive Graph of Thoughts: Test-Time Adaptive Reasoning Unifying Chain, Tree, and Graph Structures
  8. 2025A*-Thought: Efficient Reasoning via Bidirectional Compression for Low-Resource Settings