mlsanityData to 14 Oct 2026

Sharpness-Guided Group Relative Policy Optimization via Probability Shaping

arXiv 2511.00066 ↗First posted 29 Oct 2025cs.LGcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become a practical route to improve large language model reasoning, and Group Relative Policy Optimization (GRPO) is a widely used optimizer in this setting. However, RLVR training is typically performed with limited control over generalization. We revisit GRPO through a robustness-based generalization view, where the generalization loss is upper bounded by a combination of the empirical loss and a sharpness surrogate measured by the gradient norm. Building on this perspective, we propose Sharpness-Guided GRPO (GRPO-SG), a simple token-weighted variant of GRPO that downweights tokens likely to cause overly large gradients, reducing sharp updates and stabilizing optimization, thereby improving generalization. Experiments across mathematical reasoning, logic puzzles and tool-augmented question answering show consistent improvements over GRPO, along with smoother gradient-norm trajectories, supporting GRPO-SG as a simple and effective generalization-oriented upgrade to GRPO for RLVR.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 66 of its 84 references that are in the map. First posted on arXiv 29 Oct 2025.

Builds on

66 of 84 references.

Closest ancestors · 18

Same direction, newest first

  1. 2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning4 Oct
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
  4. 2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMs19 May
  5. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  6. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  7. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  8. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  9. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  10. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  11. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
6 more
  1. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  4. 2024On Designing Effective RL Reward at Training Time for LLM Reasoning19 Oct
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  12. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
36 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  2. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  4. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  5. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  8. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  9. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  11. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  12. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 533
  13. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  14. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  15. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
  16. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  17. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  18. 2017Three Factors Influencing Minima in SGDcited by 273
  19. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  20. 2025MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learningcited by 173
  21. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 172
  22. 2025Qwen2.5-1M Technical Reportcited by 164
  23. 2022Generalized Federated Learning via Sharpness Aware Minimizationcited by 144
  24. 2015On the properties of variational approximations of Gibbs posteriorscited by 121
  25. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  26. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
  27. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  28. 2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
  29. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  30. 2020Relative Flatness and Generalizationcited by 88
  31. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  32. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 58
  33. 2024TIS-DPO: Token-level Importance Sampling for Direct Preference Optimization With Estimated Weightscited by 42
  34. 2022Stochastic Multiple Target Sampling Gradient Descentcited by 14
  35. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  36. 2024Asymptotic Unbiased Sample Sampling to Speed Up Sharpness-Aware Minimizationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigationcited by 7
  2. 2026Implicit Strategic Optimization: Rethinking Long-Horizon Decision-Making in Adversarial Poker Environmentscited by 1
  3. 2026Advantage Scale Calibration Imbalance in Group-Relative Optimization under Low-Variance Rewards: Diagnosis and Bounded Recoverycited by 1
  4. 2026Stable-MM-R1: Anchoring Multimodal Reasoning Dynamics via Entropy-Guided Stratificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MCPO: Mastery-Consolidated Policy Optimization for Large Reasoning Models
  2. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
  3. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferences
  4. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  5. 2026GRPO-VPS: Enhancing Group Relative Policy Optimization with Verifiable Process Supervision for Effective Reasoning
  6. 2025SSPO: Subsentence-level Policy Optimization
  7. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
  8. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning