mlsanityData to 7 Oct 2026

Group-in-Group Policy Optimization for LLM Agent Training

arXiv 2505.10978 ↗First posted 16 May 2025cs.LG cs.AIcited by 442

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Recent advances in group-based reinforcement learning (RL) have driven frontier large language models (LLMs) in single-turn tasks like mathematical reasoning. However, their scalability to multi-turn LLM agent training remains limited. Unlike static tasks, agent-environment interactions unfold over many steps and often yield sparse or delayed rewards, making credit assignment across individual steps significantly more challenging. In this work, we propose Group-in-Group Policy Optimization (GiGPO), a novel RL algorithm that achieves fine-grained credit assignment for LLM agents while preserving the appealing properties of group-based RL: critic-free, low memory, and stable convergence. GiGPO introduces a two-level structure for estimating relative advantage: (i) At the episode-level, GiGPO computes macro relative advantages based on groups of complete trajectories; (ii) At the step-level, GiGPO introduces an anchor state grouping mechanism that retroactively constructs step-level groups by identifying repeated environment states across trajectories. Actions stemming from the same state are grouped together, enabling micro relative advantage estimation. This hierarchical structure effectively captures both global trajectory quality and local step effectiveness without relying on auxiliary models or additional rollouts. We evaluate GiGPO on challenging agent benchmarks, including ALFWorld and WebShop, as well as tool-integrated reasoning on search-augmented QA tasks, using Qwen2.5-1.5B/3B/7B-Instruct. Crucially, GiGPO delivers fine-grained per-step credit signals, achieves performance gains of > 12% on ALFWorld and > 9% on WebShop over GRPO, and obtains superior performance on QA tasks (42.1% on 3B and 47.2% on 7B): all while maintaining the same GPU memory overhead, identical LLM rollout, and incurring little to no additional time cost.

Key facts. Cited by 442 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 61 of its 83 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

61 of 83 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  2. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  3. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 53

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2025Qwen2.5-VL Technical Reportcited by 5,418
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2016OpenAI Gymcited by 1,684
  7. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  8. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  9. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  12. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  13. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  14. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  15. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  16. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  17. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  18. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  19. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  20. 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 359
  21. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 357
  22. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  23. 2025ToolRL: Reward is All Tool Learning Needscited by 342
  24. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  25. 2024CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challengescited by 278
  26. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  27. 2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 210
  28. 2025ZeroSearch: Incentivize the Search Capability of LLMs without Searchingcited by 203
  29. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  30. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  31. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  32. 2023Multimodal Web Navigation with Instruction-Finetuned Foundation Modelscited by 159
  33. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  34. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  35. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 142
  36. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
  37. 2025Reinforcement Learning for Long-Horizon Interactive LLM Agentscited by 103
  38. 2015Language Understanding for Text-based Games Using Deep Reinforcement Learningcited by 70

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 85; 2026: 357.

442 papers.

Sort:

By area:

  1. 2026SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learningcited by 252
  2. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  4. 2025WebDancer: Towards Autonomous Information Seeking Agencycited by 180
  5. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  6. 2025Agentic Reinforced Policy Optimizationcited by 166
  7. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  8. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  9. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  10. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  11. 2026SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalizationcited by 82
  12. 2025Agent Learning via Early Experiencecited by 79
38 more
  1. 2025Reinforcing Multi-Turn Reasoning in LLM Agents via Fine-Grained Reward Structure and Credit Assignmentcited by 77
  2. 2025AgentEvolver: Towards Efficient Self-Evolving Agent Systemcited by 77
  3. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 75
  4. 2025Tree Search for LLM Agent Reinforcement Learningcited by 73
  5. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 60
  6. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 58
  7. 2025Training-Free Group Relative Policy Optimizationcited by 53
  8. 2025The Path Not Taken: RLVR Provably Learns Off the Principalscited by 52
  9. 2025OneThinker: All-in-one Reasoning Model for Image and Videocited by 52
  10. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  11. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  12. 2026Hindsight Credit Assignment for Long-Horizon LLM Agentscited by 51
  13. 2025Graph-R1: Towards Agentic GraphRAG Framework via End-to-end Reinforcement Learningcited by 50
  14. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
  15. 2026Self-Distilled Agentic Reinforcement Learningcited by 49
  16. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  17. 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 45
  18. 2026SkillOS: Learning Skill Curation for Self-Evolving Agentscited by 45
  19. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  20. 2025Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agentscited by 43
  21. 2025Cognitive Kernel-Pro: A Framework for Deep Research Agents and Agent Foundation Models Trainingcited by 42
  22. 2025Scaling Agent Learning via Experience Synthesiscited by 40
  23. 2025Agentic Entropy-Balanced Policy Optimizationcited by 39
  24. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  25. 2026TCOD: Exploring Temporal Curriculum in On-Policy Distillation for Multi-turn Autonomous Agentscited by 38
  26. 2026Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Taskscited by 37
  27. 2026MetaClaw: Just Talk -- An Agent That Meta-Learns and Evolves in the Wildcited by 35
  28. 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligencecited by 35
  29. 2026Dynamic Dual-Granularity Skill Bank for Agentic RLcited by 33
  30. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  31. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 31
  32. 2025CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Criticcited by 29
  33. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learningcited by 28
  34. 2025Turn-PPO: Turn-Level Advantage Estimation with PPO for Improved Multi-Turn RL in Agentic LLMscited by 28
  35. 2026Skill1: Unified Evolution of Skill-Augmented Agents via Reinforcement Learningcited by 28
  36. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMscited by 25
  37. 2026Gen-Searcher: Reinforcing Agentic Search for Image Generationcited by 25
  38. 2026Beyond Trajectory-Level Attribution: Graph-Based Credit Assignment for Agentic Reinforcement Learningcited by 23

50 most cited of 442 All 442 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025End-to-End Optimization of LLM-Driven Multi-Agent Search Systems via Heterogeneous-Group-Based Reinforcement Learning
  2. 2025Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
  3. 2026iGRPO: Self-Feedback-Driven LLM Reasoning
  4. 2025Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning
  5. 2026Breaking textitWinner-Takes-All: Cooperative Policy Optimization Improves Diverse LLM Reasoning
  6. 2025Reinforcement Learning-Augmented LLM Agents for Collaborative Decision Making and Performance Optimization
  7. 2025Reinforced Language Models for Sequential Decision Making
  8. 2025Group Sequence Policy Optimization