# Encouraging Good Processes Without the Need for Good Answers: Reinforcement Learning for LLM Agent Planning

arXiv: https://arxiv.org/abs/2508.19598 · HTML: https://mlsanity.com/paper/2508.19598/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 28 of 28)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools](https://mlsanity.com/paper/2406.12793/) (2024)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [ReTool: Reinforcement Learning for Strategic Tool Use in LLMs](https://mlsanity.com/paper/2504.11536/) (2025)
- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [Evaluating Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2310.19736/) (2023)
- [WebDancer: Towards Autonomous Information Seeking Agency](https://mlsanity.com/paper/2505.22648/) (2025)
- [Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospects](https://mlsanity.com/paper/2401.03428/) (2024)
- [Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding](https://mlsanity.com/paper/2401.12954/) (2024)
- [Reward Shaping to Mitigate Reward Hacking in RLHF](https://mlsanity.com/paper/2502.18770/) (2025)
- [On Designing Effective RL Reward at Training Time for LLM Reasoning](https://mlsanity.com/paper/2410.15115/) (2024)
- [Process Reward Models for LLM Agents: Practical Framework and Directions](https://mlsanity.com/paper/2502.10325/) (2025)
- [AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning](https://mlsanity.com/paper/2401.05268/) (2024)
- [RRM: Robust Reward Model Training Mitigates Reward Hacking](https://mlsanity.com/paper/2409.13156/) (2024)
- [TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usage](https://mlsanity.com/paper/2308.03427/) (2023)
- [Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models](https://mlsanity.com/paper/2411.07140/) (2024)

## Built on by (most cited first, 17 of 17)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges](https://mlsanity.com/paper/2604.13602/) (2026)
- [Toward Efficient Agents: Memory, Tool learning, and Planning](https://mlsanity.com/paper/2601.14192/) (2026)
- [Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning](https://mlsanity.com/paper/2510.25992/) (2025)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [A Unified Definition of Hallucination: It's The World Model, Stupid!](https://mlsanity.com/paper/2512.21577/) (2025)
- [TodoEvolve: Learning to Architect Agent Planning Systems](https://mlsanity.com/paper/2602.07839/) (2026)
- [RLVP: Penalize the Path, Reward the Outcome](https://mlsanity.com/paper/2607.07435/) (2026)
- [Harnessing Pre-Resolution Signals for Future Prediction Agents](https://mlsanity.com/paper/2604.15719/) (2026)
- [MagicSelector: Joint Optimization for Agent Tool Selection via Counterfactual Decomposition and Progressive Reranking](https://mlsanity.com/paper/2607.17751/) (2026)
- [SpecXMaster Technical Report](https://mlsanity.com/paper/2603.23101/) (2026)
- [RECIPE: Procedural Planning via Grounding in Instructional Video](https://mlsanity.com/paper/2605.19976/) (2026)
- [Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models](https://mlsanity.com/paper/2609.03493/) (2026)
- [CityPlanner: A Sandbox Agent for Executable Urban Planning](https://mlsanity.com/paper/2609.09578/) (2026)
- [ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learning](https://mlsanity.com/paper/2610.05935/) (2026)
- [DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents](https://mlsanity.com/paper/2610.19376/) (2026)

## Abstract (arXiv)

The functionality of Large Language Model (LLM) agents is primarily determined by two capabilities: action planning and answer summarization. The former, action planning, is the core capability that dictates an agent's performance. However, prevailing training paradigms employ end-to-end, multi-objective optimization that jointly trains both capabilities. This paradigm faces two critical challenges: imbalanced optimization objective allocation and scarcity of verifiable data, making it difficult to enhance the agent's planning capability. To address these challenges, we propose Reinforcement Learning with Tool-use Rewards (RLTR), a novel framework that decouples the training process to enable a focused, single-objective optimization of the planning module. Crucially, RLTR introduces a reward signal based on tool-use completeness to directly evaluate the quality of tool invocation sequences. This method offers a more direct and reliable training signal than assessing the final response content, thereby obviating the need for verifiable data. Our experiments demonstrate that RLTR achieves an 8%-12% improvement in planning performance compared to end-to-end baselines. Moreover, this enhanced planning capability, in turn, translates to a 5%-6% increase in the final response quality of the overall agent system.