mlsanityData to 7 Oct 2026

Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning

arXiv 2602.09598 ↗First posted 10 Feb 2026cs.CLcited by 5

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Tool-integrated reasoning (TIR) enables LLM agents to solve tasks through planning, tool use, and iterative revision, but outcome-only reinforcement learning in this setting suffers from sparse, delayed rewards and weak step-level credit assignment. In long-horizon TIR trajectories, an early irrecoverable mistake can determine success or failure, making it crucial to localize the first irrecoverable step and leverage it for fine-grained credit assignment. We propose Error-Localized Policy Optimization (ELPO), which localizes the first irrecoverable step via binary-search rollout trees under a fixed rollout budget, converts the resulting tree into stable learning signals through hierarchical advantage attribution, and applies error-localized adaptive clipping to strengthen corrective updates on the critical step and its suffix. Across TIR benchmarks in math, science QA, and code execution, ELPO consistently outperforms strong Agentic RL baselines under comparable sampling budgets, with additional gains in Pass@K and Major@K scaling, rollout ranking quality, and tool-call efficiency. Our code will be publicly released soon.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 31 of its 38 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

31 of 38 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  3. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 24

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  8. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  9. 2024DeepSeek-V3 Technical Reportcited by 724
  10. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  11. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  12. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
12 more
  1. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  2. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
  3. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  4. 2025ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replaycited by 75
  5. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  6. 2025SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attributioncited by 68
  7. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  8. 2025Understanding Tool-Integrated Reasoningcited by 38
  9. 2025AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learningcited by 32
  10. 2025Demystifying Reinforcement Learning in Agentic Reasoningcited by 32
  11. 2025Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agentcited by 28
  12. 2025Towards Effective Code-Integrated Reasoningcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnessescited by 13
  2. 2026TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agentscited by 3
  3. 2026Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoningcited by 0
  4. 2026Toolcompass: Guiding Tool Trialing, Not Suppressing Itcited by 0
  5. 2026Dependency-Aware Reward Shaping for Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026HiDiffTIR: Hierarchical Difficulty-Aware Policy Optimization for Multi-Turn Tool-Integrated Reasoning
  2. 2025Agentic Reinforced Policy Optimization
  3. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  4. 2025Tool-Augmented Policy Optimization: Synergizing Reasoning and Adaptive Tool Use with Reinforcement Learning
  5. 2025PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoning
  6. 2025Multi-Agent Tool-Integrated Policy Optimization
  7. 2026Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning
  8. 2026Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spaces