mlsanityData to 7 Oct 2026

Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignment

arXiv 2505.11821 ↗First posted 17 May 2025cs.LGcited by 77

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement Learning (RL) approaches have been wildly used to enhance the reasoning capabilities of Large Language Model (LLM) agents in long-horizon, multi-turn scenarios. Such interactions can be formalized as turn-level Markov decision processes (MDPs), where intermediate rewards are often available. However, most prior work relies on sparse trajectory-level rewards, resulting in poor credit assignment, while dense turn-level rewards remain underexplored. In this paper, we investigate how to effectively leverage dense turn-level reward structures in RL algorithms, specifically Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO), to enable fine-grained credit assignment. We categorize reward structures into three types based on their granularity: (1) terminal reward; (2) delayed reward; (3) per-turn reward, each corresponding to a distinct turn-level MDP formulation, and derive GRPO and PPO algorithms tailored to each case, respectively. Experiments on multi-turn search and game agent tasks show that, for both GRPO and PPO, dense per-turn reward structures consistently outperform sparse terminal and delayed reward structures in terms of training dynamics and numerical results. Furthermore, on search tasks, PPO with dense per-turn rewards achieves greater training stability and faster convergence, and delivers the highest answer correctness among all state-of-the-art methods across diverse question-answering datasets.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 61 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

32 of 61 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning21 Apr
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Process Reward Models for LLM Agents: Practical Framework and Directions14 Feb
  6. 2025Process Reinforcement through Implicit Rewards3 Feb
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoning13 Jan
  9. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning16 Oct
  10. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  8. 2024OpenAI o1 System Cardcited by 1,220
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  10. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  11. 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 377
  12. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
9 more
  1. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  2. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 276
  3. 2025ToRL: Scaling Tool-Integrated RLcited by 166
  4. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  5. 2025Agentic Reasoning and Tool Integration for LLMs via Reinforcement Learningcited by 107
  6. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  7. 2025Acting Less is Reasoning More! Teaching Model to Act Efficientlycited by 80
  8. 2025Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoningcited by 67
  9. 2023A Survey of Temporal Credit Assignment in Deep Reinforcement Learningcited by 56

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 65.

77 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2026SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weightingcited by 38
  3. 2025Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Gamescited by 30
  4. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 29
  5. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28
  6. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Modelscited by 19
  7. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  8. 2026RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedbackcited by 16
  9. 2025MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMscited by 14
  10. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  11. 2025MTIR-SQL: Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQLcited by 10
  12. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMscited by 10
38 more
  1. 2026Dynamic Skill Lifecycle Management for Agentic Reinforcement Learningcited by 10
  2. 2026GEAR: Granularity-Adaptive Advantage Reweighting for LLM Agents via Self-Distillationcited by 10
  3. 2026UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillationcited by 10
  4. 2026Exact Is Easier: Credit Assignment for Cooperative LLM Agentscited by 9
  5. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  6. 2025VIDEOP2R: Video Understanding from Perception to Reasoningcited by 8
  7. 2026InfoPO: Information-Driven Policy Optimization for User-Centric Agentscited by 8
  8. 2026CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilitiescited by 8
  9. 2026Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimizationcited by 7
  10. 2026SeeUPO: Sequence-Level Agentic-RL with Convergence Guaranteescited by 6
  11. 2026TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agentscited by 6
  12. 2026GAGPO: Generalized Advantage Grouped Policy Optimizationcited by 6
  13. 2026PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignmentcited by 6
  14. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?cited by 5
  15. 2026TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimizationcited by 5
  16. 2026Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Modelscited by 5
  17. 2026Skill0.5: Joint Skill Internalization and Utilization for Out-of-Distribution Generalization in Agentic Reinforcement Learningcited by 5
  18. 2026TACO: Tool-Augmented Credit Optimization for Agentic Tool Usecited by 5
  19. 2025PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoningcited by 4
  20. 2026Improving Parametric Knowledge Access in Reasoning Language Modelscited by 4
  21. 2026BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agentscited by 4
  22. 2026LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoningcited by 4
  23. 2026Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifierscited by 3
  24. 2026Closing the Reflection Gap: A Free Calibration Bonus for Agentic RLcited by 3
  25. 2026Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agentscited by 3
  26. 2026Cold-Start Personalization via Training-Free Priors from Structured World Modelscited by 2
  27. 2026Harmonizing Dense and Sparse Signals in Multi-turn RL: Dual-Horizon Credit Assignment for Industrial Sales Agentscited by 2
  28. 2026MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialoguecited by 2
  29. 2026FiRe: Fine-grained Multimodal Reasoning for Enhanced Image Generationcited by 2
  30. 2026Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thoughtcited by 2
  31. 2026Beyond Trajectory Rewards: Step-level Credit Assignment for Agentic Search via Graph Modelingcited by 2
  32. 2026Scaling Scientific Discovery Environments for Turn-Level Agentic RLcited by 2
  33. 2026TL-GRPO: Turn-Level RL for Reasoning-Guided Iterative Optimizationcited by 1
  34. 2026The Auton Agentic AI Frameworkcited by 1
  35. 2026Negative Advantages Is a Double-Edged Sword: Calibrating advantages in GRPO for Search Agentscited by 1
  36. 2026Not All Turns Matter: Credit Assignment for Multi-Turn Jailbreakingcited by 1
  37. 2026Drowning in Routine: Signal Dilution in Multi-Turn Agent Trainingcited by 1
  38. 2026RLVP: Penalize the Path, Reward the Outcomecited by 1

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents
  2. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  3. 2026AT2PO: Agentic Turn-based Policy Optimization via Tree Search
  4. 2025Training Task Reasoning LLM Agents for Multi-turn Task Planning via Single-turn Reinforcement Learning
  5. 2025Agentic Reinforced Policy Optimization
  6. 2026TCPO: Turn-Level Credit Policy Optimization
  7. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMs
  8. 2026TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents