mlsanityData to 14 Oct 2026

On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization

arXiv 2505.18830 ↗First posted 24 May 2025cs.LG cs.CLcited by 31

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has become popular in enhancing the reasoning capabilities of large language models (LLMs), with Group Relative Policy Optimization (GRPO) emerging as a widely used algorithm in recent systems. Despite GRPO's widespread adoption, we identify a previously unrecognized phenomenon we term Lazy Likelihood Displacement (LLD), wherein the likelihood of correct responses marginally increases or even decreases during training. This behavior mirrors a recently discovered misalignment issue in Direct Preference Optimization (DPO), attributed to the influence of negative gradients. We provide a theoretical analysis of GRPO's learning dynamic, identifying the source of LLD as the naive penalization of all tokens in incorrect responses with the same strength. To address this, we develop a method called NTHR, which downweights penalties on tokens contributing to the LLD. Unlike prior DPO-based approaches, NTHR takes advantage of GRPO's group-based structure, using correct responses as anchors to identify influential tokens. Experiments on math reasoning benchmarks demonstrate that NTHR effectively mitigates LLD, yielding consistent performance gains across models ranging from 0.5B to 3B parameters.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 34 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

29 of 34 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  2. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  3. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  4. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 20

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 735
  9. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
  10. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  11. 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 152
  12. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
8 more
  1. 2020Neural collapse with unconstrained featurescited by 142
  2. 2024Learning Dynamics of LLM Finetuningcited by 95
  3. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  4. 2019The Local Elasticity of Neural Networkscited by 37
  5. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  6. 2024Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representationscited by 24
  7. 2025The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hackingcited by 18
  8. 2024DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Modelscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 18.

31 papers.

Sort:

By area:

  1. 2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 20
  2. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  3. 2025On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacementcited by 15
  4. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
  5. 2025AVATAR: Reinforcement Learning to See, Hear, and Reason Over Videocited by 12
  6. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 11
  7. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradientscited by 11
  8. 2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learningcited by 10
  9. 2026Robust Tool Use via Fission-GRPO: Learning to Recover from Execution Errorscited by 10
  10. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 8
  11. 2026Where Hindsight Credit Can Reside: A Signed-Capacity View of Token Updates in RLVRcited by 8
  12. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 4
19 more
  1. 2025Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learningcited by 3
  2. 2025Entropy-Guided Reasoning Compressioncited by 3
  3. 2026Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learningcited by 3
  4. 2026The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Creditscited by 3
  5. 2026STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stabilitycited by 3
  6. 2026Score Centering Stabilizes Off-policy Reinforcement Learningcited by 3
  7. 2026Improving Search Agent with One Line of Codecited by 2
  8. 2026StAR: Segment Anything Reasonercited by 2
  9. 2026Self-Evolving Deep Research via Joint Generation and Evaluationcited by 2
  10. 2025For-Value: Efficient Forward-Only Data Valuation for finetuning LLMs and VLMscited by 1
  11. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoningcited by 1
  12. 2026SCA: Segment-Wise CoT Compression with Answer Alignmentcited by 1
  13. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVRcited by 1
  14. 2026Metacognitive Behavioral Tuning of Large Language Models for Multi-Hop Question Answeringcited by 0
  15. 2026ANCORA: Learning to Question via Manifold-Anchored Self-Play for Verifiable Reasoningcited by 0
  16. 2026Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Modelscited by 0
  17. 2026Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Losscited by 0
  18. 2026GRPO Training Dynamics for Small Language Modelscited by 0
  19. 2026GAW-PO: Preference Optimization with Gradient-Aligned Token Weightscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
  2. 2026On the Hidden Objective Biases of Group-based Reinforcement Learning
  3. 2025Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting
  4. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic
  5. 2026ReCo: Reweighting GRPO Against Distributional Concentration
  6. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  7. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
  8. 2025Mitigating Think-Answer Mismatch in LLM Reasoning Through Noise-Aware Advantage Reweighting