mlsanityData to 14 Oct 2026

Understanding Diversity Collapse in RLVR via the Lens of Overtraining

arXiv 2606.15455 ↗First posted 13 Jun 2026cs.LG cs.AIcited by 10

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become a key approach for enhancing the reasoning abilities of large language models. However, RLVR often suffers from diversity collapse: Pass@1 improves while high-k Pass@k degrades, which is viewed as a narrowing of the model's reasoning boundary. We formalize this diversity collapse through the lens of overtraining: once a problem's contribution to the reference metric has effectively saturated, further updates no longer expand what the model can solve but still concentrate probability mass on the trajectories favored by on-policy sampling. Under a standard setup with few rollouts per problem, even a single observed success places a problem in a nearly saturated regime for high-k Pass@k, so most updates in standard RLVR are overtraining from the boundary perspective. This perspective also suggests a reading of whether RLVR can expand the model's reasoning abilities beyond the base model: since RLVR is structurally biased against high-k Pass@k, its aggregate decline does not by itself mean that no new reasoning gains occurred. Interventionally, restricting updates to problems with zero observed success lifts Pass@256 above the base model on difficult benchmarks; observationally, a non-trivial fraction of initially unsolvable problems become solvable during standard RLVR training. Building on these findings, we propose Bayesian Boundary Gating (BBG), which redirects optimization away from overtraining by estimating each problem's marginal contribution to the reasoning boundary. Across multiple reasoning benchmarks, BBG improves average Pass@k across a wide range of k.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 46 references that are in the map. First posted on arXiv 13 Jun 2026.

Builds on

38 of 46 references.

Closest ancestors · 22

Same direction, newest first

  1. 2025Pass@k Metric for RLVR: A Diagnostic Tool of Exploration, But Not an Objective20 Nov
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models8 Nov
  3. 2025Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models6 Nov
  4. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients27 Oct
  5. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View5 Oct
  6. 2025ExGRPO: Learning to Reason from Experience2 Oct
  7. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models2 Oct
  8. 2025No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping26 Sep
  9. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models14 Aug
  10. 2025RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization31 Jul
  11. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  12. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts2 Jun
10 more
  1. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  2. 2025Self-Evolving Curriculum for LLM Reasoning20 May
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  9. 2023Let's Verify Step by Step31 May
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 16

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2016Concrete Problems in AI Safetycited by 1,642
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  10. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  11. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  12. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
4 more
  1. 2023Goodhart's Law in Reinforcement Learningcited by 41
  2. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
  3. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  4. 2026When Sharpening Becomes Collapse: Sampling Bias and Semantic Coupling in RL with Verifiable Rewardscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Training AI Scientists to Replicate Researchcited by 9
  2. 2026When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversioncited by 3
  3. 2026Question Begets Question: Self-Evolving Curriculum for Reinforcement Fine-Tuning on Competition Mathematicscited by 2
  4. 2026BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?cited by 2
  5. 2026Phantom Gains: Auditing Self-Improvement Against a Measured Nullcited by 2
  6. 2026Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gainscited by 1
  7. 2026Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Spacecited by 0
  8. 2026Do Quantum AIs Dream in Paths? Path-Integral Slow Thinking through Grover Interferencecited by 0
  9. 2026Are Human-Aligned Models Models of Humans? A Turing-Test Gap in Preference Alignmentcited by 0
  10. 2026What Will Post-Training Fix? Per-Problem Gains Are Shared Across Independent RL Runs, and Existing Checkpoints Predict Them Better Than A Priori Signalscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Measuring and Mitigating Solution Mode Collapse in RLVR
  2. 2026Linear Dynamics in the RLVR Training of Large Language Models
  3. 2026The Unlearnability Phenomenon in RLVR for Language Models
  4. 2026Boosting LLM Exploration via Weak-Model Guidance in RLVR
  5. 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity
  6. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  7. 2025The Path Not Taken: RLVR Provably Learns Off the Principals
  8. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifiers