mlsanityData to 14 Oct 2026

Group Policy Gradient

arXiv 2510.03679 ↗First posted 4 Oct 2025cs.LG stat.MLcited by 2

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

We introduce Group Policy Gradient (GPG), a family of critic-free policy-gradient estimators for general MDPs. Inspired by the success of GRPO's approach in Reinforcement Learning from Human Feedback (RLHF), GPG replaces a learned value function with a group-based Monte Carlo advantage estimator, removing the memory, compute, and hyperparameter costs of training a critic while preserving PPO's clipped-objective structure. We prove the consistency of the GPG estimator, analyze the bias-variance tradeoffs, and demonstrate empirically that GPG matches or outperforms PPO on standard benchmarks. GPG makes better use of parallel simulations, which, together with its critic-free design, results in more efficient use of computational resources than PPO.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 10 of its 17 references that are in the map. First posted on arXiv 4 Oct 2025.

Builds on

10 of 17 references.

References · 10

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  6. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 251
  7. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 98
  8. 2020VarGrad: A Low-Variance Gradient Estimator for Variational Inferencecited by 75
  9. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  10. 2025Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimizationcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Mattercited by 0
  2. 2026GR2PO: Group Relative Return Policy Optimization for Continuous Robot Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  2. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
  3. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
  4. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
  5. 2026AGPO: Adaptive Group Policy Optimization with Dual Statistical Feedback
  6. 2026On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse
  7. 2026Constrained Group Relative Policy Optimization
  8. 2026MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following