mlsanityData to 14 Oct 2026

Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning

arXiv 2508.19598 ↗First posted 27 Aug 2025cs.LGcited by 17

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

The functionality of Large Language Model (LLM) agents is primarily determined by two capabilities: action planning and answer summarization. The former, action planning, is the core capability that dictates an agent's performance. However, prevailing training paradigms employ end-to-end, multi-objective optimization that jointly trains both capabilities. This paradigm faces two critical challenges: imbalanced optimization objective allocation and scarcity of verifiable data, making it difficult to enhance the agent's planning capability. To address these challenges, we propose Reinforcement Learning with Tool-use Rewards (RLTR), a novel framework that decouples the training process to enable a focused, single-objective optimization of the planning module. Crucially, RLTR introduces a reward signal based on tool-use completeness to directly evaluate the quality of tool invocation sequences. This method offers a more direct and reliable training signal than assessing the final response content, thereby obviating the need for verifiable data. Our experiments demonstrate that RLTR achieves an 8%-12% improvement in planning performance compared to end-to-end baselines. Moreover, this enhanced planning capability, in turn, translates to a 5%-6% increase in the final response quality of the overall agent system.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 28 of its 45 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

28 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  2. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMs15 Apr
  3. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  8. 2020Knowledge Distillation: A Surveycited by 1,603
  9. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  10. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  11. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  12. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
13 more
  1. 2024Understanding the planning of LLM agents: A surveycited by 421
  2. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
  3. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
  4. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 202
  5. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 188
  6. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 132
  7. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 117
  8. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  9. 2024On Designing Effective RL Reward at Training Time for LLM Reasoningcited by 87
  10. 2024AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planningcited by 81
  11. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 76
  12. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 55
  13. 2024Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Modelscited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 12.

17 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  4. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 16
  5. 2025Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoningcited by 11
  6. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  7. 2025A Unified Definition of Hallucination: It's The World Model, Stupid!cited by 6
  8. 2026TodoEvolve: Learning to Architect Agent Planning Systemscited by 2
  9. 2026RLVP: Penalize the Path, Reward the Outcomecited by 2
  10. 2026Harnessing Pre-Resolution Signals for Future Prediction Agentscited by 1
  11. 2026MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Rerankingcited by 1
  12. 2026SpecXMaster Technical Reportcited by 0
5 more
  1. 2026RECIPE: Procedural Planning via Grounding in Instructional Videocited by 0
  2. 2026Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Modelscited by 0
  3. 2026CityPlanner: A Sandbox Agent for Executable Urban Planningcited by 0
  4. 2026ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learningcited by 0
  5. 2026DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ToolRL: Reward is All Tool Learning Needs
  2. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use
  3. 2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Training
  4. 2025ToolExpander: Extending the Frontiers of Tool-Using Reinforcement Learning to Weak LLMs
  5. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods
  6. 2026Demystifying Reinforcement Learning for Long-Horizon Tool-Using Agents: A Comprehensive Recipe
  7. 2025Enhancing Agentic RL with Progressive Reward Shaping and Value-based Sampling Policy Optimization
  8. 2025Tool Zero: Training Tool-Augmented LLMs via Pure RL from Scratch