mlsanityData to 14 Oct 2026

Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity

arXiv 2512.05962 ↗First posted 5 Dec 2025cs.LG cs.AIcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning (RL) has become the de facto standard for tuning LLMs to solve tasks involving reasoning. However, growing evidence shows that models trained in such way often suffer from a significant loss in diversity. We argue that this arises because RL implicitly optimizes the "mode-seeking" or "zero-forcing" Reverse KL to a target distribution causing the model to concentrate mass on certain high-probability regions of the target while neglecting others. In this work, we instead begin from an explicit target distribution, obtained by filtering out incorrect answers while preserving the relative probabilities of correct ones. Starting from a pre-trained LLM, we approximate this target distribution using the α-divergence family, which unifies prior approaches and enables direct control of the precision-diversity trade-off by interpolating between mode-seeking and mass-covering divergences. On a Lean theorem-proving benchmark, our method achieves state-of-the-art performance along the coverage-precision Pareto frontier, outperforming all prior methods on the coverage axis.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 34 of its 70 references that are in the map. First posted on arXiv 5 Dec 2025.

Builds on

34 of 70 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  2. 2025Magistral12 Jun
  3. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  4. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  6. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  9. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 24

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  7. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  8. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 287
  9. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  10. 2025Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learningcited by 180
  11. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  12. 2020A Distributional Approach to Controlled Text Generationcited by 119
12 more
  1. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  2. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  3. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 82
  4. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 78
  5. 2025Challenging the Boundaries of Reasoning: An Olympiad-Level Math Benchmark for Large Language Modelscited by 70
  6. 2025Defeating the Training-Inference Mismatch via FP16cited by 66
  7. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  8. 2024InternLM2.5-StepProver: Advancing Automated Theorem Proving via Critic-Guided Searchcited by 34
  9. 2021Controlling Conditional Language Models without Catastrophic Forgettingcited by 32
  10. 2017Dropout Inference in Bayesian Neural Networks with Alpha-divergencescited by 28
  11. 2019Distributional Reinforcement Learning for Energy-Based Sequential Modelscited by 17
  12. 2024Guaranteed Generation from Large Language Modelscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Beyond Consensus: Trace-Level Synthesis in Mixture of Agentscited by 4
  2. 2026Binary Rewards and Reinforcement Learning: Fundamental Challengescited by 1
  3. 2025Learning to Reason in LLMs by Expectation Maximizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
  2. 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
  3. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning
  4. 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
  5. 2025Outcome-based Exploration for LLM Reasoning
  6. 2026R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training
  7. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning
  8. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning