mlsanityData to 7 Oct 2026

SimpleTIR: End-to-End Reinforcement Learning for Multi-Turn Tool-Integrated Reasoning

arXiv 2509.02479 ↗First posted 2 Sep 2025cs.LGcited by 145

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Large Language Models (LLMs) can significantly improve their reasoning capabilities by interacting with external tools, a paradigm known as Tool-Integrated Reasoning (TIR). However, extending TIR to multi-turn scenarios using Reinforcement Learning (RL) is often hindered by training instability and performance collapse. We identify that such instability is primarily caused by a distributional drift from external tool feedback, leading to the generation of low-probability tokens. This issue compounds over successive turns, causing catastrophic gradient norm explosions that derail the training process. To address this challenge, we introduce SimpleTIR , a plug-and-play algorithm that stabilizes multi-turn TIR training. Its core strategy is to identify and filter out trajectories containing void turns, i.e., turns that yield neither a code block nor a final answer. By removing these problematic trajectories from the policy update, SimpleTIR effectively blocks the harmful, high-magnitude gradients, thus stabilizing the learning dynamics. Extensive experiments show that SimpleTIR achieves state-of-the-art performance on challenging math reasoning benchmarks, notably elevating the AIME24 score from a text-only baseline of 22.1 to 50.5 when starting from the Qwen2.5-7B base model. Furthermore, by avoiding the constraints of supervised fine-tuning, SimpleTIR encourages the model to discover diverse and sophisticated reasoning patterns, such as self-correction and cross-validation.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 35 references that are in the map. First posted on arXiv 2 Sep 2025.

Builds on

31 of 35 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention14 Aug
  2. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  3. 2025Geometric-Mean Policy Optimization28 Jul
  4. 2025Agentic Reinforced Policy Optimization26 Jul
  5. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  6. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  7. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  8. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  10. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 18

  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  3. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  4. 2025Group Sequence Policy Optimizationcited by 788
  5. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  6. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  7. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 322
  8. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  9. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  10. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  11. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  12. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
6 more
  1. 2025Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solvingcited by 62
  2. 2025Understanding Tool-Integrated Reasoningcited by 38
  3. 2025Towards Effective Code-Integrated Reasoningcited by 23
  4. 2013Hierarchical Solution of Markov Decision Processes using Macro-actionscited by 19
  5. 2025Divergence-Augmented Policy Optimizationcited by 4
  6. 2025Logit Dynamics in Softmax Policy Gradient Methodscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 48; 2026: 97.

145 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
  3. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  4. 2025DeepAgent: A General Reasoning Agent with Scalable Toolsetscited by 69
  5. 2025Agent0: Unleashing Self-Evolving Agents from Zero Data via Tool-Integrated Reasoningcited by 68
  6. 2025In-the-Flow Agentic System Optimization for Effective Planning and Tool Usecited by 67
  7. 2025Training-Free Group Relative Policy Optimizationcited by 53
  8. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  9. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 29
  10. 2025Agent0-VL: Exploring Self-Evolving Agent for Tool-Integrated Vision-Language Reasoningcited by 28
  11. 2025Flash-Searcher: Fast and Effective Web Agents via DAG-Based Parallel Executioncited by 27
  12. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 23
38 more
  1. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 23
  2. 2026AgentOCR: Reimagining Agent History via Optical Self-Compressioncited by 22
  3. 2026On-Policy Distillation with Curriculum Turn-level Guidance for Multi-turn Agentscited by 21
  4. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 20
  5. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 17
  6. 2025GEM: A Gym for Agentic LLMscited by 17
  7. 2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 17
  8. 2025WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seekingcited by 17
  9. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  10. 2026SOD: Step-wise On-policy Distillation for Small Language Model Agentscited by 17
  11. 2025Multi-Agent Tool-Integrated Policy Optimizationcited by 16
  12. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 16
  13. 2025On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacementcited by 15
  14. 2026MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoningcited by 15
  15. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 15
  16. 2026Watching, Reasoning, and Searching: A Video Deep Research Benchmark on Open Web for Agentic Video Reasoningcited by 14
  17. 2026The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agentscited by 14
  18. 2026PyVision-RL: Forging Open Agentic Vision Models via RLcited by 14
  19. 2026ECHO: Terminal Agents Learn World Models for Freecited by 13
  20. 2026ReSkill: Reconciling Skill Creation with Policy Optimization in Agentic RLcited by 13
  21. 2026PivotRL: High Accuracy Agentic Post-Training at Low Compute Costcited by 12
  22. 2025Scaling Generalist Data-Analytic Agentscited by 11
  23. 2025GAP: Graph-Based Agent Planning with Parallel Tool Use and Reinforcement Learningcited by 11
  24. 2025InfoFlow: Reinforcing Search Agent Via Reward Density Optimizationcited by 11
  25. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  26. 2026RC-GRPO: Reward-Conditioned Group Relative Policy Optimization for Multi-Turn Tool Calling Agentscited by 11
  27. 2026ARLArena: A Unified Framework for Stable Agentic Reinforcement Learningcited by 11
  28. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  29. 2025Repurposing Synthetic Data for Fine-grained Search Agent Supervisioncited by 10
  30. 2025MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQLcited by 10
  31. 2025Asymmetric Proximal Policy Optimization: mini-critics boost LLM reasoningcited by 9
  32. 2026MatchTIR: Fine-Grained Supervision for Tool-Integrated Reasoning via Bipartite Matchingcited by 9
  33. 2025Variational Reasoning for Language Modelscited by 8
  34. 2026TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarioscited by 8
  35. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  36. 2026The Interplay of Harness Design and Post-Training in LLM Agentscited by 8
  37. 2025Reinforcement Learning for Tool-Integrated Interleaved Thinking towards Cross-Domain Generalizationcited by 7
  38. 2026TIDE: Trajectory-based Diagnostic Evaluation of Test-Time Improvement in LLM Agentscited by 7

50 most cited of 145 All 145 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning
  2. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  3. 2025Tool-Augmented Policy Optimization: Synergizing Reasoning and Adaptive Tool Use with Reinforcement Learning
  4. 2026Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning
  5. 2025RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use
  6. 2025Acting Less is Reasoning More! Teaching Model to Act Efficiently
  7. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
  8. 2026ASTER: Agentic Scaling with Tool-integrated Extended Reasoning