# Learning from the Irrecoverable: Error-Localized Policy Optimization for Tool-Integrated LLM Reasoning

arXiv: https://arxiv.org/abs/2602.09598 · HTML: https://mlsanity.com/paper/2602.09598/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 31 of its 38 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [DeepSeek-V3 Technical Report](https://mlsanity.com/paper/2412.19437/) (2024)
- [Group-in-Group Policy Optimization for LLM Agent Training](https://mlsanity.com/paper/2505.10978/) (2025)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025)
- [REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization](https://mlsanity.com/paper/2501.03262/) (2025)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [ToRL: Scaling Tool-Integrated RL](https://mlsanity.com/paper/2503.23383/) (2025)
- [Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learning](https://mlsanity.com/paper/2505.01441/) (2025)
- [Acting Less is Reasoning More! Teaching Model to Act Efficiently](https://mlsanity.com/paper/2504.14870/) (2025)
- [Tree Search for LLM Agent Reinforcement Learning](https://mlsanity.com/paper/2509.21240/) (2025)
- [ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay](https://mlsanity.com/paper/2505.16282/) (2025)
- [SPA-RL: Reinforcing LLM Agents via Stepwise Progress Attribution](https://mlsanity.com/paper/2505.20732/) (2025)
- [Agentic Entropy-Balanced Policy Optimization](https://mlsanity.com/paper/2510.14545/) (2025)
- [Understanding Tool-Integrated Reasoning](https://mlsanity.com/paper/2508.19201/) (2025)
- [AutoTIR: Autonomous Tools Integrated Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2507.21836/) (2025)
- [Demystifying Reinforcement Learning in Agentic Reasoning](https://mlsanity.com/paper/2510.11701/) (2025)
- [Reinforced Internal-External Knowledge Synergistic Reasoning for Efficient Adaptive Search Agent](https://mlsanity.com/paper/2505.07596/) (2025)

## Built on by (most cited first, 6 of 6)

- [HarnessCompass: Guiding Automatic Harness Evolution toward Generalizable and Effective Agent Harnesses](https://mlsanity.com/paper/2608.01918/) (2026)
- [TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents](https://mlsanity.com/paper/2605.09934/) (2026)
- [Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning](https://mlsanity.com/paper/2605.05226/) (2026)
- [Toolcompass: Guiding Tool Trialing, Not Suppressing It](https://mlsanity.com/paper/2609.25678/) (2026)
- [Dependency-Aware Reward Shaping for Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.01207/) (2026)
- [Failure Recovery via Belief Updates and World Modeling for Long-Horizon Agents](https://mlsanity.com/paper/2610.18082/) (2026)

## Abstract (arXiv)

Tool-integrated reasoning (TIR) enables LLM agents to solve tasks through planning, tool use, and iterative revision, but outcome-only reinforcement learning in this setting suffers from sparse, delayed rewards and weak step-level credit assignment. In long-horizon TIR trajectories, an early irrecoverable mistake can determine success or failure, making it crucial to localize the first irrecoverable step and leverage it for fine-grained credit assignment. We propose Error-Localized Policy Optimization (ELPO), which localizes the first irrecoverable step via binary-search rollout trees under a fixed rollout budget, converts the resulting tree into stable learning signals through hierarchical advantage attribution, and applies error-localized adaptive clipping to strengthen corrective updates on the critical step and its suffix. Across TIR benchmarks in math, science QA, and code execution, ELPO consistently outperforms strong Agentic RL baselines under comparable sampling budgets, with additional gains in Pass@K and Major@K scaling, rollout ranking quality, and tool-call efficiency. Our code will be publicly released soon.