mlsanityData to 7 Oct 2026

Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends

arXiv 2509.24203 ↗First posted 29 Sep 2025cs.LG cs.AI cs.CLcited by 13

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Off-policy reinforcement learning (RL) for large language models (LLMs) is attracting growing interest, driven by practical constraints in real-world applications, the complexity of LLM-RL infrastructure, and the need for further innovations of RL methodologies. While classic REINFORCE and its modern variants like Group Relative Policy Optimization (GRPO) are typically regarded as on-policy algorithms with limited tolerance of off-policyness, we present in this work a first-principles derivation for group-relative REINFORCE -- a REINFORCE variant that uses the within-group mean reward as the baseline for advantage calculation -- without assuming a specific training data distribution, showing that it admits a native off-policy interpretation. This perspective yields two general principles for adapting REINFORCE to truly off-policy settings: regularizing policy updates, and actively shaping the data distribution. Our analysis demystifies some myths about the roles of importance sampling and clipping in GRPO, unifies and reinterprets two recent algorithms -- Online Policy Mirror Descent and Asymmetric REINFORCE -- as regularized forms of the REINFORCE loss, and offers theoretical justification for seemingly heuristic data-weighting strategies. Our findings lead to actionable insights that are validated with extensive empirical studies, and open up new opportunities for principled algorithm design in off-policy RL for LLMs. Source code for this work is available at https://github.com/agentscope-ai/Trinity-RFT/tree/main/examples/rec_gsm8k.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 70 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

21 of 70 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Models10 Sep
  2. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning11 Aug
  3. 2025Geometric-Mean Policy Optimization28 Jul
  4. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
  5. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning10 Jun
  6. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library6 Jun
  7. 2025Learning to Reason under Off-Policy Guidance21 Apr
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework20 May
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 10

  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2024OpenAI o1 System Cardcited by 1,220
  3. 2015Prioritized Experience Replaycited by 1,070
  4. 2025Group Sequence Policy Optimizationcited by 788
  5. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  6. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 121
  7. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  8. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  9. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  10. 2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 19
  2. 2026R3L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplificationcited by 12
  3. 2026An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shiftcited by 2
  4. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  5. 2026Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Modelscited by 1
  6. 2026SCA: Segment-Wise CoT Compression with Answer Alignmentcited by 1
  7. 2026Skip-Connected Policy Optimization for Implicit Advantagecited by 1
  8. 2026F-TIS: Harnessing Diverse Models in Collaborative GRPOcited by 1
  9. 2025Do Math Reasoning LLMs Help Predict the Impact of Public Transit Events?cited by 0
  10. 2026POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensemblescited by 0
  11. 2026Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learningcited by 0
  12. 2026Staleness-Learning Rate Scaling Laws for Asynchronous RLHFcited by 0
1 more
  1. 2026Learning Stateful Predictive Knowledge From Experiencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training
  2. 2026On the Hidden Objective Biases of Group-based Reinforcement Learning
  3. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  4. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  5. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  6. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment
  7. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  8. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback