mlsanityData to 13 Oct 2026

Progress- and Reliability-Oriented Group Policy Optimization for Agentic Reinforcement Learning

arXiv 2607.04242 ↗First posted 5 Jul 2026cs.AIcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Group-based reinforcement learning (RL) has become an effective paradigm for improving large language model agents on long-horizon interactive tasks. To obtain finer-grained policy updates than trajectory-level optimization, recent work has moved toward step-level group-based RL, where intermediate steps are grouped and compared within a rollout batch. However, step-level advantage estimation is sensitive to how groups are formed: grouping by broad state keys improves coverage but may compare actions taken under different histories, while enforcing historical consistency yields fairer comparisons at the cost of fragmented groups and missing peer-comparison signal. In this paper, we propose ProGPO (Progress- and Reliability-Oriented Group Policy Optimization), a learned-critic-free method for context-consistent step-level learning. ProGPO keeps exact-prefix action comparison, and complements sparse peer comparisons with transition credit derived from rollout-based state potentials. To estimate these potentials reliably, ProGPO combines semantic expansion with inverse-variance fusion across history depths. We evaluate ProGPO on two challenging agentic tasks, ALFWorld and WebShop, with Qwen2.5-1.5B-Instruct. Results show that ProGPO improves over matched agentic RL baselines under comparable computational overhead, and additional Qwen2.5-3B-Instruct experiments further test the scalability of the proposed method.

Key facts. Cited by 2 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 37 references that are in the map. First posted on arXiv 5 Jul 2026.

Builds on

31 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks26 Feb
  2. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  3. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning24 Apr
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2017Proximal Policy Optimization Algorithmscited by 14,592
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,502
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,117
  9. 2017Deep reinforcement learning from human preferencescited by 3,871
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,691
  11. 2023Let's Verify Step by Stepcited by 3,672
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,022
14 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,729
  2. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,321
  3. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,052
  4. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,953
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,904
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,399
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,242
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,222
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,219
  10. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,167
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 767
  12. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 551
  13. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 211
  14. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 160

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learningcited by 0
  2. 2026AdaStep: Adaptive Step Credit Weighting for Agentic Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks
  2. 2026Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning
  3. 2026PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks
  4. 2026PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs
  5. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  6. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learning
  7. 2026T2SPO: Trajectory-to-Step Policy Optimization for Agentic Reinforcement Learning
  8. 2026Rethinking Groups in Critic-Free RLVR