Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning
On the map: LLM reasoning → Reasoning for agents, planning and code
also in 1 more
Abstract (arXiv)
Tool-integrated reasoning (TIR) offers a direct way to extend thinking models beyond the limits of text-only reasoning. Paradoxically, we observe that tool-enabled evaluation can degrade reasoning performance even when the strong thinking models make almost no actual tool calls. In this paper, we investigate how to inject natural tool-use behavior into a strong thinking model without sacrificing its no-tool reasoning ability, and present a comprehensive TIR recipe. We highlight that (i) the effectiveness of TIR supervised fine-tuning (SFT) hinges on the learnability of teacher trajectories, which should prioritize problems inherently suited for tool-augmented solutions; (ii) controlling the proportion of tool-use trajectories could mitigate the catastrophic forgetting of text-only reasoning capacity; (iii) optimizing for pass@k and response length instead of training loss could maximize TIR SFT gains while preserving headroom for reinforcement learning (RL) exploration; (iv) a stable RL with verifiable rewards (RLVR) stage, built upon suitable SFT initialization and explicit safeguards against mode collapse, provides a simple yet remarkably effective solution. When applied to Qwen3 thinking models at 4B and 30B scales, our recipe yields models that achieve state-of-the-art performance in a wide range of benchmarks among open-source models, such as 96.7% and 99.2% on AIME 2025 for 4B and 30B, respectively.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 39 of its 51 references that are in the map. First posted on arXiv 7 May 2026.
Builds on
39 of 51 references.
Closest ancestors · 7
Same direction, newest first
- 2026ASTER: Agentic Scaling with Tool-integrated Extended Reasoning1 Feb
- 2026Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees13 Jan
- 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoning20 Nov
- 2025Demystifying Reinforcement Learning in Agentic Reasoning13 Oct
- 2025Understanding Tool-Integrated Reasoning26 Aug
- 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar
Other references · 32
- 2023GPT-4 Technical Reportcited by 18,132
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
- 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
- 2025Process Reinforcement through Implicit Rewardscited by 478
- 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
- 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
- 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
- 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
20 more
- 2025TTRL: Test-Time Reinforcement Learningcited by 253
- 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
- 2025Agentic Reinforced Policy Optimizationcited by 166
- 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
- 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 115
- 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
- 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
- 2025JustRL: Scaling a 1.5B LLM with a Simple RL Recipecited by 57
- 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
- 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 43
- 2026FrontierScience: Evaluating AI's Ability to Perform Expert-Level Scientific Taskscited by 42
- 2025From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Onescited by 40
- 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 38
- 2025Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervisioncited by 22
- 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Usecited by 16
- 2025CoRT: Code-integrated Reasoning within Thinkingcited by 15
- 2025Generalizable End-to-End Tool-Use RL with Synthetic CodeGymcited by 14
- 2025P1: Mastering Physics Olympiads with Reinforcement Learningcited by 7
- 2026P1-VL: Bridging Visual Perception and Scientific Reasoning in Physics Olympiadscited by 7
- 2025Dynamic Vocabulary Pruning: Stable LLM-RL by Taming the Tailcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Acting Less is Reasoning More! Teaching Model to Act Efficiently
- 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
- 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
- 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
- 2026TInR: Exploring Tool-Internalized Reasoning in Large Language Models
- 2025Thinking Isn't an Illusion: Overcoming the Limitations of Reasoning Models via Tool Augmentations
- 2026DeepTool: Scaling Interleaved Deliberation in Tool-Integrated Reasoning via Process-Supervised Reinforcement Learning
- 2026Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning