mlsanityData to 13 Oct 2026

Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic

arXiv 2603.01162 ↗First posted 1 Mar 2026cs.LG stat.MLcited by 21

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Group relative policy optimization (GRPO), a core methodological component of DeepSeekMath and DeepSeek-R1, has emerged as a cornerstone for scaling reasoning capabilities of large language models. Despite its widespread adoption and the proliferation of follow-up works, the theoretical properties of GRPO remain less studied. This paper provides a unified framework to understand GRPO through the lens of classical U-statistics. We demonstrate that the GRPO policy gradient is inherently a U-statistic, allowing us to characterize its mean squared error (MSE), derive the finite-sample error bound and asymptotic distribution of the suboptimality gap for its learned policy. Our findings reveal that GRPO is asymptotically equivalent to an oracle policy gradient algorithm -- one with access to a value function that quantifies the goodness of its learning policy at each training iteration -- and achieves asymptotically optimal performance within a broad class of policy gradient algorithms. Furthermore, we establish a universal scaling law that offers principled guidance for selecting the optimal group size. Empirical experiments further validate our theoretical findings, demonstrating that the optimal group size is universal, and verify the oracle property of GRPO.

Key facts. Cited by 21 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 95 of its 145 references that are in the map. First posted on arXiv 1 Mar 2026.

Builds on

95 of 145 references.

Closest ancestors · 28

Same direction, newest first

  1. 2026Your Group-Relative Advantage Is Biased13 Jan
  2. 2025TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models9 Dec
  3. 2025Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards5 Nov
  4. 2025What is the objective of reasoning with reinforcement learning?15 Oct
  5. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training1 Oct
  6. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends29 Sep
  7. 2025Single-stream Policy Optimization16 Sep
  8. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  9. 2025RePO: Replay-Enhanced Policy Optimization11 Jun
  10. 2025BNPO: Beta Normalization Policy Optimization3 Jun
  11. 2025On-Policy RL with Optimal Reward Baseline29 May
  12. 2025ARM: Adaptive Reasoning Model26 May
16 more
  1. 2025Stable Reinforcement Learning for Efficient Reasoning23 May
  2. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  3. 2025Learning to Reason under Off-Policy Guidance21 Apr
  4. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM19 Apr
  5. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  6. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce15 Apr
  7. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  8. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  9. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  10. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  13. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  14. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  15. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  16. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 67

  1. 2017Attention Is All You Needcited by 60,106
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
  8. 2017Deep reinforcement learning from human preferencescited by 3,871
  9. 2015Trust Region Policy Optimizationcited by 2,513
  10. 2021Convex optimizationcited by 2,488
  11. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,223
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,003
38 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,889
  2. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,710
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,531
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2025Group Sequence Policy Optimizationcited by 806
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 657
  7. 2020MOPO: Model-based Offline Policy Optimizationcited by 632
  8. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 465
  9. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 432
  10. 2020Is Pessimism Provably Efficient for Offline RL?cited by 363
  11. 2025ToolRL: Reward is All Tool Learning Needscited by 350
  12. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 339
  13. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  14. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  15. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 277
  16. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 268
  17. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 229
  18. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  19. 2023Nash Learning from Human Feedbackcited by 223
  20. 2019A Theoretical Analysis of Deep Q-Learningcited by 212
  21. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 186
  22. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 161
  23. 2024Token-level Direct Preference Optimizationcited by 135
  24. 2024Provably Robust DPO: Aligning Language Models with Noisy Feedbackcited by 134
  25. 2019Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Samplingcited by 118
  26. 2022A Review of Off-Policy Evaluation in Reinforcement Learningcited by 94
  27. 2022Settling the Sample Complexity of Model-Based Offline Reinforcement Learningcited by 83
  28. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  29. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  30. 2020Batch Policy Learning in Average Reward Markov Decision Processescited by 69
  31. 2016Estimating Dynamic Treatment Regimes in Mobile Health Using V-learningcited by 65
  32. 2025Parallel-R1: Towards Parallel Thinking via Reinforcement Learningcited by 59
  33. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learningcited by 45
  34. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 32
  35. 2021Estimating Optimal Infinite Horizon Dynamic Treatment Regimes via pT-Learningcited by 26
  36. 2022Towards Deployment-Efficient Reinforcement Learning: Lower Bound and Optimalitycited by 24
  37. 2025What is the Alignment Objective of GRPO?cited by 23
  38. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 23

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 21.

21 papers.

Sort:

By area:

  1. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Taskscited by 8
  2. 2026DGPO: Distribution Guided Policy Optimization for Fine Grained Credit Assignmentcited by 7
  3. 2026PAIR: Prefix-Aware Internal Reward Model for Multi-Turn Agent Optimizationcited by 5
  4. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  5. 2026PACEvolve++: Improving Test-time Learning for Evolutionary Search Agentscited by 4
  6. 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoningcited by 3
  7. 2026Perturbation is All You Need for Extrapolating Language Modelscited by 1
  8. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcited by 1
  9. 2026PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thoughtcited by 1
  10. 2026PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVRcited by 1
  11. 2026Learning Perturbations to Extrapolate Your LLMcited by 0
  12. 2026Value-Gradient Hypothesis of RL for LLMscited by 0
9 more
  1. 2026CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoningcited by 0
  2. 2026Learning to Trigger: Reinforcement Learning at the Large Hadron Collidercited by 0
  3. 2026Entropy Pacing Policy Optimization for Multi-Task Agentic Reinforcement Learningcited by 0
  4. 2026Cross-View Correspondence Is a Measurement Intervention: Two-Sided Validation for Agent Evaluation and Credit Assignmentcited by 0
  5. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 0
  6. 2026ABC-Align: Prediction-Powered Alignment with Adaptive Bias Controlcited by 0
  7. 2026Sample What You Say: Aligning Language Models to Sample the Distributions They Statecited by 0
  8. 2026Prompts Live on an Arc: Gaussian Curricula in Fisher--Rao Coordinates for Rollout-Efficient GRPOcited by 0
  9. 2026Reward-Driven Learning under Prompt-Level Differential Privacycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  2. 2025GTPO: Stabilizing Group Relative Policy Optimization via Gradient and Entropy Control
  3. 2025Composing Policy Gradients and Prompt Optimization for Language Model Programs
  4. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
  5. 2026GRPO Training Dynamics for Small Language Models
  6. 2025Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning
  7. 2025GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
  8. 2025Geometric-Mean Policy Optimization