mlsanityData to 7 Oct 2026

Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning

arXiv 2511.19942 ↗First posted 25 Nov 2025cs.LGcited by 19

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to diversity collapse, where outputs lack variety. Prior work has proposed a range of heuristics to counteract this effect, but these methods are ad hoc: they frequently trade off correctness for diversity, their effectiveness varies across tasks, and in some cases they even contradict one another. In this work, we place these observations on a rigorous foundation. We first provide a formal proof of why RL fine-tuning exhibits diversity collapse via a selection and reinforcement bias. Next, we make a key observation that any reward modification to address diversity collapse only needs to be applied on the correct trajectories. Building directly on this analysis, we introduce a principled method -- differential smoothing -- that provably improves both correctness and diversity, outperforming vanilla RL as well as widely used entropy-based heuristics. Our theory precisely characterizes when existing heuristics help and why they fail, while showing that differential smoothing is universally superior. Extensive experiments with models from 1B to 7B parameters, across domains including CountDown and real-world mathematical reasoning, demonstrate consistent gains. Differential smoothing improves both Pass@1 and Pass@k, with up to 6.7% improvements on AIME24 dataset.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 53 references that are in the map. First posted on arXiv 25 Nov 2025.

Builds on

40 of 53 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Outcome-based Exploration for LLM Reasoning8 Sep
  2. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  3. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  4. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  5. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  6. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  7. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  8. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  9. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  11. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
3 more
  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 25

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2023Mistral 7Bcited by 3,065
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2024OpenAI o1 System Cardcited by 1,220
  9. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  10. 2022Large Language Models Can Self-Improvecited by 656
  11. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  12. 2025Skywork Open Reasoner 1 Technical Reportcited by 161
13 more
  1. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  2. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  3. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  4. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  5. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  6. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  7. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 82
  8. 2023Language Model Self-improvement by Reinforcement Learning Contemplationcited by 75
  9. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
  10. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 36
  11. 2025One-shot Entropy Minimizationcited by 32
  12. 2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoningcited by 28
  13. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 19.

19 papers.

Sort:

By area:

  1. 2026Maximum Likelihood Reinforcement Learningcited by 29
  2. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 27
  3. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rarecited by 10
  4. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  5. 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversitycited by 6
  6. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVRcited by 5
  7. 2026Teacher-Free Self-Training Amplifies but Does Not Compound: A Pass@K Crossover on a Free-Verifier Domaincited by 4
  8. 2026Few-Step Diffusion Language Models via Trajectory Self-Distillationcited by 3
  9. 2026When are likely answers right? On Sequence Probability and Correctness in LLMscited by 3
  10. 2026Bifrost: Steering Strategic Trajectories to Bridge Contextual Gaps for Self-Improving Agentscited by 2
  11. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priorscited by 2
  12. 2026Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learningcited by 1
7 more
  1. 2026Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolutioncited by 1
  2. 2026STATe-of-Thoughts: Structured Action Templates for Tree-of-Thoughtscited by 0
  3. 2026Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Spacecited by 0
  4. 2026Group Adaptive Clipping Policy Optimizationcited by 0
  5. 2026SAGE: Mitigating Long-Horizon Reasoning Biases via Topological Guidancecited by 0
  6. 2026Strategically Diverse Sampling for Self-Trainingcited by 0
  7. 2026Sample What You Say: Aligning Language Models to Sample the Distributions They Statecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity
  2. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
  3. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtraining
  4. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
  5. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning
  6. 2025Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs
  7. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
  8. 2026Many Ways to Succeed: Diversity-Driven RL Fine-Tuning for VLA Generalization