mlsanityData to 14 Oct 2026

Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training

arXiv 2505.22257 ↗First posted 28 May 2025cs.LG stat.MLcited by 34

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We revisit Group Relative Policy Optimization (GRPO) in both on-policy and off-policy optimization regimes. Our motivation comes from recent work on off-policy Proximal Policy Optimization (PPO), which improves training stability, sampling efficiency, and memory usage. In addition, a recent analysis of GRPO suggests that estimating the advantage function with off-policy samples could be beneficial. Building on these observations, we adapt GRPO to the off-policy setting. We show that both on-policy and off-policy GRPO objectives yield an improvement in the reward. This result motivates the use of clipped surrogate objectives in the off-policy version of GRPO. We then compare the empirical performance of reinforcement learning with verifiable rewards in post-training using both GRPO variants. Our results show that off-policy GRPO either significantly outperforms or performs on par with its on-policy counterpart.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 19 of its 27 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

19 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  10. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  11. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 40
  12. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
2 more
  1. 2025What is the Alignment Objective of GRPO?cited by 23
  2. 2024Transductive Off-policy Proximal Policy Optimizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 25.

34 papers.

Sort:

By area:

  1. 2025KnowRL: Exploring Knowledgeable Reinforcement Learning for Factualitycited by 20
  2. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 13
  3. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 12
  4. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPOcited by 10
  5. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 9
  6. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoningcited by 8
  7. 2026TSR: Trajectory-Search Rollouts for Multi-Turn RL of LLM Agentscited by 7
  8. 2025POPI: Personalizing LLMs via Optimized Natural Language Preference Inferencecited by 6
  9. 2026Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradientscited by 5
  10. 2025PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awarenesscited by 4
  11. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoningcited by 4
  12. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
22 more
  1. 2026On the Hidden Objective Biases of Group-based Reinforcement Learningcited by 3
  2. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuningcited by 3
  3. 2026Find, Fix, Reason: Context Repair for Video Reasoningcited by 3
  4. 2026FedMOA: Federated GRPO for Personalized Reasoning LLMs under Heterogeneous Rewardscited by 2
  5. 2026Beyond KL Divergence: Policy Optimization with Flexible Bregman Divergences for LLM Reasoningcited by 2
  6. 2026From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Groundingcited by 2
  7. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  8. 2026Staleness-Learning Rate Scaling Laws for Asynchronous RLHFcited by 2
  9. 2026REGEN: Replay-recycling for Expert-to-Generalist distillation with Offline Reinforcement Learningcited by 2
  10. 2025OraPO: Oracle-educated Reinforcement Learning for Data-efficient and Factual Radiology Report Generationcited by 1
  11. 2026Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMscited by 1
  12. 2026Consensus Group Relative Policy Optimization for Text Generationcited by 1
  13. 2026F-TIS: Harnessing Diverse Models in Collaborative GRPOcited by 1
  14. 2026Smaller Models are Natural Explorers for Policy-Level Diversity in GRPOcited by 1
  15. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 1
  16. 2025Multi-Reward GRPO Fine-Tuning for De-biasing Large Language Models: A Study Based on Chinese-Context Discrimination Datacited by 0
  17. 2026HiPP-Prune: Hierarchical Preference-Conditioned Structured Pruning for Vision-Language Modelscited by 0
  18. 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVRcited by 0
  19. 2026Gradient Extrapolation-Based Policy Optimizationcited by 0
  20. 2026Beyond the Literal: Decomposing Pragmatic Intent in Multimodal Meme Understandingcited by 0
  21. 2026Reason in the Words You Speak: Idiolectal Paraphrasing Off-Policy Traces for Reasoning Distillation in VideoLLMscited by 0
  22. 2026Self-Evolving Algorithm-Design Agents: Escaping In-Context Evolutionary Stagnation via Population-Curated Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  2. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  3. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
  4. 2026GOPO: Policy Optimization using Ranked Rewards
  5. 2026Constrained Group Relative Policy Optimization
  6. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems
  7. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  8. 2026RRPO: Reference-Relative Policy Optimization with Stratified Conditional Rollouts