Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement Learning (RL) enhances LLM reasoning, yet a paradox emerges as models scale: strong base models saturate standard benchmarks (e.g., MATH), yielding correct but homogeneous solutions. In such environments, the lack of failure cases causes the advantage signal in group-relative algorithms (e.g., GRPO) to vanish, driving policies into mode collapse. To address this, we propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration. Unlike standard sampling that follows model biases, CUTS flattens the local optimization landscape by sampling uniformly from constrained high-confidence candidates. We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance. Experiments on Qwen3 models demonstrate that our approach prevents policy degeneration and significantly boosts out-of-domain generalization. Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO, validating that maintaining diversity within the semantic manifold is critical for rigorous reasoning.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 17 references that are in the map. First posted on arXiv 20 Apr 2026.
Builds on
15 of 17 references.
Closest ancestors · 5
Same direction, newest first
- 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning26 Jan
- 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning17 Dec
- 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
- 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 10
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 228
- 2025Self-Rewarding Vision-Language Model via Reasoning Decompositioncited by 97
- 2025One Token to Fool LLM-as-a-Judgecited by 60
- 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
- 2025AdaReasoner: Adaptive Reasoning Enables More Flexible Thinking in Large Language Modelscited by 17
- 2025Dissecting Logical Reasoning in LLMs: A Fine-Grained Evaluation and Supervision Studycited by 17
- 2025CLUE: Non-parametric Verification from Experience via Hidden-State Clusteringcited by 12
- 2025Stable and Efficient Single-Rollout RL for Multimodal Reasoningcited by 8
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity
- 2026Back to Basics: Revisiting Exploration in Reinforcement Learning for LLM Reasoning via Generative Probabilities
- 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
- 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce
- 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management
- 2026R^3: Replay, Reflection, and Ranking Rewards for LLM Reinforcement Learning
- 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
- 2025Reasoning with Sampling: Your Base Model is Smarter Than You Think