mlsanityData to 14 Oct 2026

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

arXiv 2608.29188 ↗First posted 29 Aug 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved across all checkpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x--16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 -> 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, we find that while surface prompting fails to recover diversity, entrance-targeted interventions succeed: late-layer parameter interpolation with early checkpoints increases solution coverage by 37% at no loss in pass@1. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT--DPO--RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room. Code: https://github.com/ershiyidian/early-branch-locking.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 40 of its 43 references that are in the map. First posted on arXiv 29 Aug 2026.

Builds on

40 of 43 references.

Closest ancestors · 22

Same direction, newest first

  1. 2026BODHI: Do LLMs Branch Out and Discover Heterogeneous Inferences?3 Aug
  2. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtraining13 Jun
  3. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVR27 May
  4. 2026Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs16 Jan
  5. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning25 Nov
  6. 2025Pass@k Metric for RLVR: A Diagnostic Tool of Exploration, But Not an Objective20 Nov
  7. 2025MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model13 Oct
  8. 2025Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries9 Oct
  9. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models2 Oct
  10. 2025Diversity-Incentivized Exploration for Versatile Reasoning30 Sep
  11. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs25 Sep
  12. 2025Outcome-based Exploration for LLM Reasoning8 Sep
10 more
  1. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning20 Jun
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  3. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  6. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoning29 May
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  8. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  9. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  10. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 18

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  9. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  10. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 97
  11. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  12. 2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
6 more
  1. 2024Forking Paths in Neural Text Generationcited by 34
  2. 2025Thought Branches: Interpreting LLM Reasoning Requires Resamplingcited by 16
  3. 2025Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generationcited by 12
  4. 2025Temporal Sampling for Forgotten Reasoning in LLMscited by 12
  5. 2026Where does output diversity collapse in post-training?cited by 11
  6. 2025Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinkingcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When RLVR Shrinks the Reasoning Boundary: Diagnosing Pass@k Inversion
  2. 2026From Base Rollouts to RL Reasoning: A Budgeted Search Perspective
  3. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs
  4. 2025Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models
  5. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards
  6. 2026SCOPE-RL: Optimizing Reasoning Paths Before and After Success
  7. 2025BroRL: Scaling Reinforcement Learning via Broadened Exploration
  8. 2026Reasoning Depth and Environment Complexity: A Controlled Study of RLVR Data Allocation across Logical Reasoning Tasks