mlsanityData to 7 Oct 2026

ReTool: Reinforcement Learning for Strategic Tool Use in LLMs

arXiv 2504.11536 ↗First posted 15 Apr 2025cs.CL cs.AIcited by 377

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

While reasoning models (e.g., DeepSeek R1) trained with reinforcement learning (RL), excel in textual reasoning, they struggle in scenarios requiring structured problem-solving, such as geometric reasoning, concise computation, or complex equation solving-areas where computational tools like code interpreters (CI) demonstrate distinct advantages. To bridge this gap, we propose ReTool, which enhances long-form reasoning with tool-integrated learning, including two key features: (1) dynamic interleaving of real-time code execution within natural language reasoning processes, and (2) an automated RL paradigm that allows policy rollouts with multi-turn real-time code execution and teaches the model in learning when and how to invoke tools based on outcome feedback. ReTool employs a systematic training framework, beginning with synthetic cold-start data generation to produce code-augmented long-form reasoning traces for fine-tuning base models. Subsequent RL training leverages task outcomes as rewards to iteratively refine the model's tool use strategy, enabling autonomous discovery of optimal tool invocation patterns without human priors. Experiments on the challenging MATH Olympiad benchmark AIME demonstrate ReTool's superiority: Our 32B model achieves 67% accuracy with 400 training steps, outperforming text-based RL baseline (40% accuracy, 1080 steps) in efficiency and performance. Remarkably, ReTool-32B attains 72.5% accuracy in extended settings, surpassing OpenAI's o1-preview by 27.9%. Further analysis reveals emergent behaviors such as code self-correction, signaling an ''aha moment'' in which the model autonomously masters adaptive tool use. These findings highlight the promise of outcome-driven tool integration for advancing complex mathematical reasoning and offer new insights into hybrid neuro-symbolic systems.

Key facts. Cited by 377 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 17 of its 25 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

17 of 25 references.

References · 17

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  8. 2025s1: Simple test-time scalingcited by 1,311
  9. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2023MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuningcited by 483
  12. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 379
5 more
  1. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  2. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  3. 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
  4. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  5. 2024MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Seriescited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 175; 2026: 202.

377 papers.

Sort:

By area:

  1. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 320
  2. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  5. 2025Agentic Reinforced Policy Optimizationcited by 166
  6. 2025Thyme: Think Beyond Imagescited by 160
  7. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 145
  8. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 129
  9. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  10. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 119
  11. 2025DeepEyesV2: Toward Agentic Multimodal Modelcited by 119
  12. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
38 more
  1. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 94
  2. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 92
  3. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  4. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 88
  5. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 84
  6. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  7. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  8. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
  9. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 75
  10. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  11. 2025PyVision: Agentic Vision with Dynamic Toolingcited by 71
  12. 2025DeepAgent: A General Reasoning Agent with Scalable Toolsetscited by 69
  13. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  14. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 67
  15. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  16. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
  17. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  18. 2025Visual Agentic Reinforcement Fine-Tuningcited by 53
  19. 2025Training-Free Group Relative Policy Optimizationcited by 53
  20. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  21. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  22. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Usecited by 46
  23. 2025SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam?cited by 45
  24. 2025Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modelingcited by 44
  25. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 43
  26. 2025Training Proactive and Personalized LLM Agentscited by 42
  27. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  28. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  29. 2025Understanding Tool-Integrated Reasoningcited by 38
  30. 2026HarnessX: A Composable, Adaptive, and Evolvable Agent Harness Foundrycited by 37
  31. 2025Seed-Prover 1.5: Mastering Undergraduate-Level Theorem Proving via Learning from Experiencecited by 36
  32. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  33. 2025VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selectioncited by 34
  34. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  35. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learningcited by 32
  36. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 32
  37. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 31
  38. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 31

50 most cited of 377 All 377 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving
  2. 2025ToRL: Scaling Tool-Integrated RL
  3. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  4. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  5. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
  6. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  7. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning
  8. 2026Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning