mlsanityData to 14 Oct 2026

ToRL: Scaling Tool-Integrated RL

arXiv 2503.23383 ↗First posted 30 Mar 2025cs.CLcited by 169

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

We introduce ToRL (Tool-Integrated Reinforcement Learning), a framework for training large language models (LLMs) to autonomously use computational tools via reinforcement learning. Unlike supervised fine-tuning, ToRL allows models to explore and discover optimal strategies for tool use. Experiments with Qwen2.5-Math models show significant improvements: ToRL-7B reaches 43.3% accuracy on AIME~24, surpassing reinforcement learning without tool integration by 14% and the best existing Tool-Integrated Reasoning (TIR) model by 17%. Further analysis reveals emergent behaviors such as strategic tool invocation, self-regulation of ineffective code, and dynamic adaptation between computational and analytical reasoning, all arising purely through reward-driven learning.

Key facts. Cited by 169 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 21 references that are in the map. First posted on arXiv 30 Mar 2025.

Builds on

12 of 21 references.

References · 12

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  5. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  7. 2025LIMO: Less is More for Reasoningcited by 498
  8. 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
  9. 2024O1 Replication Journey: A Strategic Progress Report -- Part 1cited by 143
  10. 2025LIMR: Less is More for RL Scalingcited by 139
  11. 2024O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?cited by 92
  12. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 90; 2026: 79.

169 papers.

Sort:

By area:

  1. 2025ToolRL: Reward is All Tool Learning Needscited by 362
  2. 2025WebThinker: Empowering Large Reasoning Models with Deep Research Capabilitycited by 335
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  4. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  5. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
  6. 2025Agentic Reinforced Policy Optimizationcited by 172
  7. 2025RM-R1: Reward Modeling as Reasoningcited by 154
  8. 2025SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoningcited by 146
  9. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  10. 2025Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoningcited by 91
  11. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 85
  12. 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RLcited by 80
38 more
  1. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 79
  2. 2025Tree Search for LLM Agent Reinforcement Learningcited by 79
  3. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 75
  4. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 72
  5. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  6. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  7. 2025Visual Agentic Reinforcement Fine-Tuningcited by 55
  8. 2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
  9. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 54
  10. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  11. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  12. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 48
  13. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 48
  14. 2025SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam?cited by 47
  15. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 47
  16. 2025Understanding Tool-Integrated Reasoningcited by 38
  17. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learningcited by 33
  18. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 33
  19. 2025SenseNova-MARS: Empowering Multimodal Agentic Reasoning and Search via Reinforcement Learningcited by 33
  20. 2025MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool usecited by 32
  21. 2025SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agentscited by 31
  22. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 31
  23. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  24. 2025CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimizationcited by 26
  25. 2025Nemotron-Math: Efficient Long-Context Distillation of Mathematical Reasoning from Multi-Mode Supervisioncited by 26
  26. 2025Time-R1: Towards Comprehensive Temporal Reasoning in LLMscited by 25
  27. 2025Simple o3: Towards Interleaved Vision-Language Reasoningcited by 24
  28. 2025Towards Effective Code-Integrated Reasoningcited by 23
  29. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  30. 2026OmniGAIA: Towards Native Omni-Modal AI Agentscited by 22
  31. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 21
  32. 2025RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoningcited by 20
  33. 2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agentscited by 20
  34. 2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 19
  35. 2026Co-Harness: Co-Evolving Harnesses and Model Weights for LLM Agentscited by 18
  36. 2025ToolRM: Outcome Reward Models for Tool-Calling Large Language Modelscited by 17
  37. 2025Toward Effective Tool-Integrated Reasoning via Self-Evolved Preference Learningcited by 17
  38. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17

50 most cited of 169 All 169 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch
  2. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  3. 2025Tool-Augmented Policy Optimization: Synergizing Reasoning and Adaptive Tool Use with Reinforcement Learning
  4. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning
  5. 2025Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning
  6. 2026Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning
  7. 2024StepTool: Enhancing Multi-Step Tool Usage in LLMs via Step-Grained Reinforcement Learning
  8. 2026ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning