mlsanityData to 7 Oct 2026

EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity

arXiv 2507.21848 ↗First posted 29 Jul 2025cs.AIcited by 41

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large Language Models (LLMs) have made remarkable progress in enhancing step-by-step reasoning through reinforcement learning. However, the Group Relative Policy Optimization (GRPO) algorithm, which relies on sparse reward rules, often encounters the issue of identical rewards within groups, leading to the advantage collapse problem. Existing works typically address this challenge from two perspectives: enforcing model reflection to enhance response diversity, and introducing internal feedback to augment the training signal (advantage). In this work, we begin by analyzing the limitations of model reflection and investigating the policy entropy of responses at the fine-grained sample level. Based on our experimental findings, we propose the EDGE-GRPO algorithm, which adopts Entropy-Driven Advantage and Guided Error Correction to effectively mitigate the problem of advantage collapse. Extensive experiments on several main reasoning benchmarks demonstrate the effectiveness and superiority of our approach. It is available at https://github.com/ZhangXJ199/EDGE-GRPO.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 31 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

27 of 31 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  3. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025Process Reinforcement through Implicit Rewards3 Feb
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 16

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2025s1: Simple test-time scalingcited by 1,311
  4. 2024OpenAI o1 System Cardcited by 1,220
  5. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  6. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  7. 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 315
  8. 2025R1-Zero's "Aha Moment" in Visual Reasoning on a 2B Non-SFT Modelcited by 163
  9. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 134
  10. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMscited by 115
  11. 2025Concise Reasoning via Reinforcement Learningcited by 78
  12. 2025TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoningcited by 59
4 more
  1. 2025SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learningcited by 53
  2. 2025Think Twice: Enhancing LLM Reasoning by Scaling Multi-round Test-time Thinkingcited by 46
  3. 2025One-shot Entropy Minimizationcited by 32
  4. 2025Efficient Reasoning Through Suppression of Self-Affirmation Reflections in Large Reasoning Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 28.

41 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspectivecited by 50
  3. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  4. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Trainingcited by 12
  5. 2025Heterogeneous Adaptive Policy Optimization: Tailoring Optimization to Every Token's Naturecited by 11
  6. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMscited by 10
  7. 2025VIDEOP2R: Video Understanding from Perception to Reasoningcited by 8
  8. 2025FastGRPO: Accelerating Policy Optimization via Concurrency-aware Speculative Decoding and Online Draft Learningcited by 7
  9. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  10. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  11. 2025PEAR: Phase Entropy Aware Reward for Efficient Reasoningcited by 5
  12. 2026CLIPO: Contrastive Learning in Policy Optimization Generalizes RLVRcited by 5
29 more
  1. 2026StaRPO: Stability-Augmented Reinforcement Policy Optimizationcited by 5
  2. 2025M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimizationcited by 4
  3. 2026EBPO: Empirical Bayes Shrinkage for Stabilizing Group-Relative Policy Optimizationcited by 4
  4. 2026Compress the Easy, Explore the Hard: Difficulty-Aware Entropy Regularization for Efficient LLM Reasoningcited by 4
  5. 2025Verifying Meta-Awareness via Predictive Rewards in Reasoning Modelscited by 3
  6. 2026Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoningcited by 3
  7. 2026Which Tokens Matter? Adaptive Token Selection for RLVR with the Relative Surprisal Indexcited by 3
  8. 2026Smooth Operator: Smooth Verifiable Reward Activates Spatial Reasoning Ability of Vision-Language Modelcited by 2
  9. 2026HEAL: Hindsight Entropy-Assisted Learning for Reasoning Distillationcited by 2
  10. 2026Leveraging Error Diversity in Group Rollouts for Reinforcement Learningcited by 2
  11. 2026Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groupscited by 2
  12. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 2
  13. 2025The Role of Entropy in Visual Grounding: Analysis and Optimizationcited by 1
  14. 2026Smaller Models are Natural Explorers for Policy-Level Diversity in GRPOcited by 1
  15. 2026Hint-Guided Diversified Policy Optimization for LLM Reasoningcited by 1
  16. 2026Negative Self-Distillation: Learning to Reason by Avoiding Flawscited by 1
  17. 2025Boosting Process-Correct CoT Reasoning by Modeling Solvability of Multiple-Choice QAcited by 0
  18. 2026ShapE-GRPO: Shapley-Enhanced Reward Allocation for Multi-Candidate LLM Trainingcited by 0
  19. 2026Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularizationcited by 0
  20. 2026Hidden States Know Where Reasoning Diverges: Credit Assignment via Span-Level Wasserstein Distancecited by 0
  21. 2026CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agentscited by 0
  22. 2026SALT: When More Rollouts Don't Help in Group-Based Policy Optimization and How to Make Them Mattercited by 0
  23. 2026Consistency as Inductive Bias: Learning Cross-View Invariance for Robust Multimodal Reasoningcited by 0
  24. 2026When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPOcited by 0
  25. 2026ERR+: Sequential Entropy Resolution for Efficient and Decisive LLM Reasoningcited by 0
  26. 2026Spurious Advantage Hidden in GRPOcited by 0
  27. 2026Video-HopChain: Multi-Hop Questions and Confidence-Gated Exploration for Video Reasoning Modelscited by 0
  28. 2026VAA-CSEC: Vote-guided Advantage Allocation for Chinese Semantic Error Correctioncited by 0
  29. 2026Advancing Entropy-Level Credit Assignment in RLVR via Proximal Entropy Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
  2. 2026Group Entropy-Controlled Policy Optimization
  3. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO
  4. 2026GRPO Training Dynamics for Small Language Models
  5. 2025CoRPO: Adding a Correctness Bias to GRPO Improves Generalization
  6. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
  7. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models
  8. 2026Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models