Thinking Isn't an Illusion: Overcoming the Limitations of Reasoning Models via Tool Augmentations
On the map: LLM reasoning → Test-time compute and efficient thinking
also in 1 more
Abstract (arXiv)
Large Reasoning Models (LRMs) have become a central focus in today's large language model (LLM) research, where models are designed to output a step-by-step thinking process before arriving at a final answer to handle complex reasoning tasks. Despite their promise, recent empirical studies (e.g., [Shojaee et al., 2025] from Apple) suggest that this thinking process may not actually enhance reasoning ability, where LLMs without explicit reasoning actually outperform LRMs on tasks with low or high complexity. In this work, we revisit these findings and investigate whether the limitations of LRMs persist when tool augmentations are introduced. We incorporate two types of tools, Python interpreters and scratchpads, and evaluate three representative LLMs and their LRM counterparts on Apple's benchmark reasoning puzzles. Our results show that, with proper tool use, LRMs consistently outperform their non-reasoning counterparts across all levels of task complexity. These findings challenge the recent narrative that reasoning is an illusion and highlight the potential of tool-augmented LRMs for solving complex problems.
Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 14 of its 44 references that are in the map. First posted on arXiv 23 Jul 2025.
Builds on
14 of 44 references.
Closest ancestors · 4
Same direction, newest first
- 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity7 Jun
- 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models20 Mar
- 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling10 Feb
- 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs30 Dec
Other references · 10
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025Qwen3 Technical Reportcited by 8,820
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
- 2024DeepSeek-V3 Technical Reportcited by 811
- 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
- 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
Built on by
Citing papers per year: 2025: 5; 2026: 4.
9 papers.
- 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
- 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Countingcited by 18
- 2026OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scalingcited by 5
- 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questionscited by 4
- 2025Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generationcited by 2
- 2026Certifiable Boolean Reasoning Is Universalcited by 2
- 2025Fundamental Limits of Crystalline Equivariant Graph Neural Networks: A Circuit Complexity Perspectivecited by 0
- 2026Statistical Guarantees for Reasoning Probes on Looped Boolean Circuitscited by 0
- 2026REPOT: Recoverable Program-of-Thought via Checkpoint Repaircited by 0
- 2026REPOT: Recoverable Program-of-Thought via Checkpoint Repair28 May 2026
- 2026Certifiable Boolean Reasoning Is Universal4 Feb 2026
- 2026Statistical Guarantees for Reasoning Probes on Looped Boolean Circuits3 Feb 2026
- 2026OPT-Engine: Benchmarking the Limits of LLMs in Optimization Modeling via Complexity Scaling9 Jan 2026
- 2025Fundamental Limits of Crystalline Equivariant Graph Neural Networks: A Circuit Complexity Perspective7 Oct 2025
- 2025Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting6 Oct 2025
- 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey2 Sep 2025
- 2025Entropy-Guided Loop: Achieving Reasoning through Uncertainty-Aware Generation26 Aug 2025
- 2025Too Easily Fooled? Prompt Injection Breaks LLMs on Frustratingly Simple Multiple-Choice Questions16 Aug 2025
Similar, not cited
Close in wording, no citation either way
- 2025Fast, Slow, and Tool-augmented Thinking for LLMs: A Review
- 2025Exploring the System 1 Thinking Capability of Large Reasoning Models
- 2025From Proof to Program: Characterizing Tool-Induced Reasoning Hallucinations in Large Language Models
- 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- 2025Reasoning LLMs are Wandering Solution Explorers
- 2025THINK-Bench: Evaluating Thinking Efficiency and Chain-of-Thought Quality of Large Reasoning Models
- 2022Towards Reasoning in Large Language Models: A Survey
- 2025Understanding Tool-Integrated Reasoning