mlsanityData to 14 Oct 2026

Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

arXiv 2605.21125 ↗First posted 20 May 2026cs.LGcited by 7

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO), a prominent algorithm within the Reinforcement Learning from Verifiable Rewards (RLVR) framework, has achieved strong results in improving the reasoning capabilities of large language models (LLMs). However, GRPO is prone to advantage collapse, a failure mode where homogeneous rewards within a group (e.g., all correct or all incorrect answers) yield near-zero advantages and vanishing gradients. To address this, we introduce the Advantage Collapse Rate (ACR), the first diagnostic metric quantifying the proportion of training batches with ineffective gradients. Across models from 0.5B to 14B parameters on mathematical reasoning benchmarks, we show that ACR strongly predicts training stagnation and final performance. We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples, guided by real-time ACR monitoring, to enable learning from homogeneous groups without additional model rollouts. AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales, while maintaining generalization on the evaluated out-of-domain task. Code and datasets are available at https://github.com/hexixiang/Advantage-Collapse-Rate.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 50 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

42 of 50 references.

Closest ancestors · 21

Same direction, newest first

  1. 2026Your Group-Relative Advantage Is Biased13 Jan
  2. 2025M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization15 Dec
  3. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping29 Oct
  4. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning22 Oct
  5. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  6. 2025VL Norm: Rethink Loss Aggregation in RLVR9 Sep
  7. 2025DCPO: Dynamic Clipping Policy Optimization2 Sep
  8. 2025Decomposing the Entropy-Performance Exchange: The Missing Keys to Unlocking Effective Reinforcement Learning4 Aug
  9. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity29 Jul
  10. 2025Maximizing Confidence Alone Improves Reasoning28 May
  11. 2025Learning to Reason without External Rewards26 May
  12. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
9 more
  1. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  3. 2025Adaptive Group Policy Optimization: Towards Stable Training and Token-Efficient Reasoning20 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  6. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  7. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  8. 2023Let's Verify Step by Step31 May
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  10. 2017Deep Reinforcement Learning that Matterscited by 754
  11. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  12. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scalingcited by 77
9 more
  1. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  2. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practicescited by 41
  3. 2025On Entropy Control in LLM-RL Algorithmscited by 40
  4. 2025Principled Data Selection for Alignment: The Hidden Risks of Difficult Examplescited by 37
  5. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
  6. 2025Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Testscited by 19
  7. 2025Improving the Scaling Laws of Synthetic Data with Deliberate Practicecited by 17
  8. 2025Hybrid Group Relative Policy Optimization: A Multi-Sample Approach to Enhancing Policy Optimizationcited by 15
  9. 2025AlphaPO: Reward Shape Matters for LLM Alignmentcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 1
  2. 2026Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agentscited by 0
  3. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0
  4. 2026Test-time Reinforcement Learning for Anomalous Video Understandingcited by 0
  5. 2026EAGER: Enhancing Generative Event Extraction via Reinforcement Learning with Verifiable Rewardscited by 0
  6. 2026Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RLcited by 0
  7. 2026Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025NGRPO: Negative-enhanced Group Relative Policy Optimization
  2. 2025Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning
  3. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
  4. 2025AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin
  5. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  6. 2026GDEPO: Group Dual-dynamic and Equal-right Advantage Policy Optimization with Enhanced Training Data Utilization for Sample-Constrained Reinforcement Learning
  7. 2026Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
  8. 2026iGRPO: Self-Feedback-Driven LLM Reasoning