# Understanding Tool-Integrated Reasoning

arXiv: https://arxiv.org/abs/2508.19201 · HTML: https://mlsanity.com/paper/2508.19201/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 26 Aug 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Tool learning and function calling](https://mlsanity.com/agents/tool-learning-and-function-calling/)

## Builds on (most cited first, 17 of 17)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models](https://mlsanity.com/paper/2410.07985/) (2024)
- [WebSailor: Navigating Super-human Reasoning for Web Agent](https://mlsanity.com/paper/2507.02592/) (2025)
- [ToRL: Scaling Tool-Integrated RL](https://mlsanity.com/paper/2503.23383/) (2025)
- [The Invisible Leash: Why RLVR May or May Not Escape Its Origin](https://mlsanity.com/paper/2507.14843/) (2025)
- [Towards Effective Code-Integrated Reasoning](https://mlsanity.com/paper/2505.24480/) (2025)
- [Agents Play Thousands of 3D Video Games](https://mlsanity.com/paper/2503.13356/) (2025)

## Built on by (most cited first, 30 of 38)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning](https://mlsanity.com/paper/2509.02479/) (2025)
- [Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning](https://mlsanity.com/paper/2511.16043/) (2025)
- [Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning](https://mlsanity.com/paper/2509.25300/) (2025)
- [Single-stream Policy Optimization](https://mlsanity.com/paper/2509.13232/) (2025)
- [Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning](https://mlsanity.com/paper/2510.23038/) (2025)
- [AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agent](https://mlsanity.com/paper/2512.20745/) (2025)
- [Cogito, Ergo Ludo: An Agent that Learns to Play by Reasoning and Planning](https://mlsanity.com/paper/2509.25052/) (2025)
- [Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning](https://mlsanity.com/paper/2509.22601/) (2025)
- [MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQL](https://mlsanity.com/paper/2510.25510/) (2025)
- [MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matching](https://mlsanity.com/paper/2601.10712/) (2026)
- [Autoformalizer with Tool Feedback](https://mlsanity.com/paper/2510.06857/) (2025)
- [Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalization](https://mlsanity.com/paper/2510.11184/) (2025)
- [Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluation](https://mlsanity.com/paper/2602.09305/) (2026)
- [Nested Browser-Use Learning for Agentic Information Seeking](https://mlsanity.com/paper/2512.23647/) (2025)
- [FS-Researcher: Test-Time Scaling for Long-Horizon Research Tasks with File-System-Based Agents](https://mlsanity.com/paper/2602.01566/) (2026)
- [Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning](https://mlsanity.com/paper/2602.09598/) (2026)
- [E3-TIR: Enhanced Experience Exploitation for Tool-Integrated Reasoning](https://mlsanity.com/paper/2604.09455/) (2026)
- [The Tool-Overuse Illusion: Why Does LLM Prefer External Tools over Internal Knowledge?](https://mlsanity.com/paper/2604.19749/) (2026)
- [ASTER: Agentic Scaling with Tool-integrated Extended Reasoning](https://mlsanity.com/paper/2602.01204/) (2026)
- [PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoning](https://mlsanity.com/paper/2510.26020/) (2025)
- [CoSineVerifier: Tool-Augmented Answer Verification for Computation-Oriented Scientific Questions](https://mlsanity.com/paper/2512.01224/) (2025)
- [ChartAgent: A Chart Understanding Framework with Tool Integrated Reasoning](https://mlsanity.com/paper/2512.14040/) (2025)
- [Agent Explorative Policy Optimization for Multimodal Agentic Reasoning](https://mlsanity.com/paper/2605.28774/) (2026)
- [MuSEAgent: A Multimodal Reasoning Agent with Stateful Experiences](https://mlsanity.com/paper/2603.27813/) (2026)
- [Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning](https://mlsanity.com/paper/2604.05404/) (2026)
- [When to Trust Tools? Adaptive Tool Trust Calibration For Tool-Integrated Math Reasoning](https://mlsanity.com/paper/2604.08281/) (2026)
- [Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning](https://mlsanity.com/paper/2605.06326/) (2026)
- [PruneTIR: Inference-Time Tool Call Pruning for Effective yet Efficient Tool-Integrated Reasoning](https://mlsanity.com/paper/2605.09931/) (2026)
- [FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions](https://mlsanity.com/paper/2509.17177/) (2025)

## Abstract (arXiv)

We study why Tool-Integrated Reasoning (TIR) makes Large Language Models (LLMs) more capable. While LLMs integrated with tools like Python code interpreters show great promise, a principled theory explaining why this paradigm is effective has been missing. This work provides the first formal proof that TIR fundamentally expands an LLM's capabilities. We demonstrate that tools enable a strict expansion of the model's empirical and feasible support, breaking the capability ceiling of pure-text models by unlocking problem-solving strategies that are otherwise impossible or intractably verbose. To guide model behavior without compromising training stability and performance, we also introduce Advantage Shaping Policy Optimization (ASPO), a novel algorithm that directly modifies the advantage function to guide the policy behavior. We conduct comprehensive experiments on challenging mathematical benchmarks, leveraging a Python interpreter as the external tool. Our results show that the TIR model decisively outperforms its pure-text counterpart on the pass@k metric. Crucially, this advantage is not confined to computationally-intensive problems but extends to those requiring significant abstract insight. We further identify the emergent cognitive patterns that illustrate how models learn to think with tools. Finally, we report improved tool usage behavior with early code invocation and much more interactive turns with ASPO. Overall, our work provides the first principled explanation for TIR's success, shifting the focus from the mere fact that tools work to why and how they enable more powerful reasoning.