mlsanityData to 7 Oct 2026

Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RL

arXiv 2607.01490 ↗First posted 1 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning post-training dramatically improves LLM reasoning, but suffers from training instability and diversity collapse. Advantage functions offer an appealing fix: they reshape the training objective, reweight which rollouts drive learning, and are trivial to implement. Yet a proliferation of methods makes it unclear which advantage to use and when. We cut through the confusion with a unifying framework that decomposes any advantage into its positive and negative gradient mass along two orthogonal axes. On the sign axis, imbalanced updates collapse either entropy or weight geometry. On the difficulty axis, hard-problem focus sharpens signal but costs sample size. Both trade-offs shift during training: exploration favors balance and hard focus; exploitation favors suppression and medium focus. This motivates FADE (Focal Advantage with Dynamic Entropy), a self-adapting advantage that reads training dynamics to schedule the gradient weight automatically. FADE reaches peak pass@1 20k steps earlier than the best static baseline at the 7B scale and 2k steps earlier at the 32B , while achieving the best accuracy-diversity trade-off across all pass@k on LiveCodeBench and AIME.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 60 of its 74 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

60 of 74 references.

Closest ancestors · 24

Same direction, newest first

  1. 2026On the Implicit Reward Overfitting and the Low-rank Dynamics in RLVR7 May
  2. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models10 Apr
  3. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-training24 Feb
  4. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare6 Feb
  5. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learning30 Jan
  6. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Exploration26 Jan
  7. 2026Your Group-Relative Advantage Is Biased13 Jan
  8. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients27 Oct
  9. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Think16 Oct
  10. 2025What is the objective of reasoning with reinforcement learning?15 Oct
  11. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Models1 Oct
  12. 2025Clip-Low Increases Entropy and Clip-High Decreases Entropy in Reinforcement Learning of Large Language Models30 Sep
12 more
  1. 2025Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning29 Sep
  2. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models29 Sep
  3. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  4. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  5. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  6. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  11. 2023Let's Verify Step by Step31 May
  12. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 36

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  9. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  10. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  11. 2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 165
  12. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
24 more
  1. 2023TACO: Topics in Algorithmic COde generation datasetcited by 150
  2. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  3. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  4. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  5. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  6. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 87
  7. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  8. 2016Variational inference for Monte Carlo objectivescited by 83
  9. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  10. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  11. 2024Variational Best-of-N Alignmentcited by 49
  12. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
  13. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 40
  14. 2023Scaling laws for single-agent reinforcement learningcited by 39
  15. 2018Actor-Critic Policy Optimization in Partially Observable Multiagent Environmentscited by 35
  16. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  17. 2026Maximum Likelihood Reinforcement Learningcited by 29
  18. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  19. 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 25
  20. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 24
  21. 2026Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Modelcited by 6
  22. 2026AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learningcited by 5
  23. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholdscited by 2
  24. 2026The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Creditscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
  2. 2026When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVR
  3. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
  4. 2026On Advantage Estimates for Max@K Policy Gradients
  5. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
  6. 2026Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training
  7. 2026Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
  8. 2026Control-Diverse Reinforcement Fine-Tuning: Decoupling the Shared Control Bottleneck of RL Post-Training