mlsanityData to 7 Oct 2026

Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification

arXiv 2503.06639 ↗First posted 9 Mar 2025cs.LG stat.MLcited by 108

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO) was introduced and used recently for promoting reasoning in LLMs under verifiable (binary) rewards. We show that the mean + variance calibration of these rewards induces a weighted contrastive loss in which the contrastive samples are synthetic data drawn from the previous policy. While GRPO was originally paired with clipping to keep updates near the old policy, we analyze variants that differ in reward normalization (mean-only vs mean + variance) and in how they regularize updates using KL divergence: either penalizing divergence from the previous model (mirror), penalizing divergence from a fixed reference model πref, or combining both forms of regularization. For each, the optimal policy πn admits an explicit form in terms of the binary reward and the first and second order statistics of the reward under πn-1, as well as the policies πn-1 and πref. Iterating results in a sequence πn whose probability of success (PoS) obeys a simple recurrence that converges to a fixed point determined by the reference PoS and the regularization strength. We further show that this fixed point exceeds the reference, demonstrating that GRPO amplifies the policy's probability of success.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 26 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

12 of 26 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 9

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2024DeepSeek-V3 Technical Reportcited by 724
  4. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  5. 2020Mirror Descent Policy Optimizationcited by 102
  6. 2025What is the Alignment Objective of GRPO?cited by 23
  7. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Modelscited by 22
  8. 2024Apple Intelligence Foundation Language Modelscited by 22
  9. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexitycited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 48; 2026: 60.

108 papers.

Sort:

By area:

  1. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  2. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learningcited by 100
  3. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 67
  4. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 52
  5. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 40
  6. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 35
  7. 2025Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoningcited by 34
  8. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Trainingcited by 33
  9. 2025MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learningcited by 29
  10. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 27
  11. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 23
  12. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
38 more
  1. 2025Improving Discrete Diffusion Unmasking Policies Beyond Explicit Reference Policiescited by 21
  2. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learningcited by 19
  3. 2025VeriReason: Reinforcement Learning with Testbench Feedback for Reasoning-Enhanced Verilog Generationcited by 16
  4. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  5. 2025Skywork-R1V3 Technical Reportcited by 16
  6. 2025Loong: Synthesize Long Chain-of-Thoughts at Scale through Verifierscited by 14
  7. 2025EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Frameworkcited by 13
  8. 2025CTRLS: Chain-of-Thought Reasoning via Latent State-Transitioncited by 13
  9. 2026Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scalingcited by 12
  10. 2026WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoningcited by 12
  11. 2025LegalDelta: Enhancing Legal Reasoning in LLMs via Reinforcement Learning with Chain-of-Thought Guided Information Gaincited by 11
  12. 2025Student-Centered Distillation Narrows the Agentic Gap Between Small and Large LLMscited by 11
  13. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoningcited by 11
  14. 2025Revoking Amnesia: RL-based Trajectory Optimization to Resurrect Erased Concepts in Diffusion Modelscited by 11
  15. 2025Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradientscited by 11
  16. 2025RLMR: Reinforcement Learning with Mixed Rewards for Creative Writingcited by 10
  17. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradientscited by 10
  18. 2026AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluationcited by 10
  19. 2025VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RLcited by 9
  20. 2026When Right Meets Wrong: Bilateral Context Conditioning with Reward-Confidence Correction for GRPOcited by 9
  21. 2026Learning to Hint for Reinforcement Learningcited by 9
  22. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoningcited by 8
  23. 2026Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVRcited by 8
  24. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 8
  25. 2025RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMscited by 7
  26. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  27. 2025Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impactcited by 6
  28. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 6
  29. 2026LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learningcited by 6
  30. 2025From Emergence to Control: Probing and Modulating Self-Reflection in Language Modelscited by 5
  31. 2025Rethinking Reward Signals in Video GRPO: When Scores Become Targetscited by 5
  32. 2025Pentest-R1: Towards Autonomous Penetration Testing Reasoning Optimized via Two-Stage Reinforcement Learningcited by 4
  33. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoningcited by 4
  34. 2025CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal Reasoningcited by 4
  35. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4
  36. 2026Beyond Binary: Turning Partial Success into Dense Verifiable Rewards for Reinforcement Learning in Code Generationcited by 3
  37. 2026Asymmetric Prompt Weighting for Reinforcement Learning with Verifiable Rewardscited by 3
  38. 2026InfoSFT: Learn More and Forget Less with Information-Aware Token Weightingcited by 3

50 most cited of 108 All 108 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  2. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare
  3. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening
  4. 2026A Unified Framework for Rethinking Policy Divergence Measures in GRPO
  5. 2026Discounted Beta-Bernoulli Reward Estimation for Sample-Efficient Reinforcement Learning with Verifiable Rewards
  6. 2026GOPO: Policy Optimization using Ranked Rewards
  7. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  8. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences