mlsanityData to 14 Oct 2026

GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models

arXiv 2504.09696 ↗First posted 13 Apr 2025cs.CLcited by 82

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Group Relative Policy Optimization (GRPO), which is widely adopted by R1-like reasoning models, has advanced mathematical reasoning. Nevertheless, GRPO faces challenges in reward sparsity, verbosity, and inadequate focus on problem difficulty. We propose GRPO-LEAD, enhancing GRPO with: (1) length-regularized rewards to encourage conciseness while maintaining accuracy; (2) explicit penalties for incorrect solutions to improve model precision; and (3) difficulty-aware advantage reweighting for robust generalization on challenging problems. Comprehensive evaluations demonstrate that GRPO-LEAD significantly improves reasoning accuracy, conciseness, and efficiency. Our approach achieves state-of-the-art performance for 14B-scale models, underscoring the synergy of our methods with appropriate model scale and high-quality data. Our source code, generated dataset, and models are available at https://github.com/aeroplanepaper/GRPO-LEAD.

Key facts. Cited by 82 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 13 of its 21 references that are in the map. First posted on arXiv 13 Apr 2025.

Builds on

13 of 21 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  2. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  3. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach26 Mar
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 7

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  4. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  5. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  6. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  7. 2017Reinforcement Learning with a Corrupted Reward Channelcited by 70

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 45; 2026: 37.

82 papers.

Sort:

By area:

  1. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2025Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMscited by 117
  4. 2025Geometric-Mean Policy Optimizationcited by 111
  5. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 89
  6. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 62
  7. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoningcited by 56
  8. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  9. 2025Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoningcited by 35
  10. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 33
  11. 2025Promoting Efficient Reasoning with Verifiable Stepwise Rewardcited by 28
  12. 2025Difficulty-Aware Agentic Orchestration for Query-Specific Multi-Agent Workflowscited by 27
38 more
  1. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  2. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 23
  3. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  4. 2026Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulationcited by 19
  5. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 19
  6. 2025SATORI-R1: Incentivizing Multimodal Reasoning through Explicit Visual Anchoringcited by 17
  7. 2025DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimizationcited by 16
  8. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  9. 2026Devil is in Narrow Policy: Unleashing Exploration in Driving VLA Modelscited by 15
  10. 2026Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learningcited by 13
  11. 2025Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generationcited by 11
  12. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  13. 2025Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoningcited by 11
  14. 2026PAPO: Stabilizing Rubric Integration Training via Decoupled Advantage Normalizationcited by 11
  15. 2025Decomposing Elements of Problem Solving: What "Math" Does RL Teach?cited by 10
  16. 2025SEM: Reinforcement Learning for Search-Efficient Large Language Modelscited by 9
  17. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  18. 2025Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoningcited by 7
  19. 2025RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMscited by 7
  20. 2025SmartThinker: Learning to Compress and Preserve Reasoning by Step-Level Length Controlcited by 7
  21. 2025ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interactioncited by 7
  22. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  23. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  24. 2025REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Reasoningcited by 6
  25. 2025StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Modelscited by 6
  26. 2025e1: Learning Adaptive Control of Reasoning Effortcited by 6
  27. 2026Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policycited by 6
  28. 2025From Sufficiency to Reflection: Reinforcement-Guided Thinking Quality in Retrieval-Augmented Reasoning for LLMscited by 5
  29. 2025VL Norm: Rethink Loss Aggregation in RLVRcited by 4
  30. 2025Explore Briefly, Then Decide: Mitigating LLM Overthinking via Cumulative Entropy Regulationcited by 4
  31. 2025TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevancecited by 4
  32. 2025Towards Flash Thinking via Decoupled Advantage Policy Optimizationcited by 4
  33. 2025PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMscited by 4
  34. 2025Reasoner for Real-World Event Detection: Scaling Reinforcement Learning via Adaptive Perplexity-Aware Sampling Strategycited by 3
  35. 2025Verifying Meta-Awareness via Predictive Rewards in Reasoning Modelscited by 3
  36. 2026DIVA-GRPO: Enhancing Multimodal Reasoning through Difficulty-Adaptive Variant Advantagecited by 3
  37. 2026Sim2Act: Robust Simulation-to-Decision Learning via Adversarial Calibration and Group-Relative Perturbationcited by 3
  38. 2026A Deep Dive into Scaling RL for Code Generation with Synthetic Data and Curriculacited by 3

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?
  2. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
  3. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning
  4. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  5. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
  6. 2026GRPO Training Dynamics for Small Language Models
  7. 2026AMIR-GRPO: Inducing Implicit Preference Signals into GRPO
  8. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity