mlsanityData to 7 Oct 2026

The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models

arXiv 2510.02230 ↗First posted 2 Oct 2025cs.AI cs.CL cs.CVcited by 17

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key method for improving Large Language Models' reasoning capabilities, yet recent evidence suggests it may paradoxically shrink the reasoning boundary rather than expand it. This paper investigates the shrinkage issue of RLVR by analyzing its learning dynamics and reveals two critical phenomena that explain this failure. First, we expose negative interference in RLVR, where learning to solve certain training problems actively reduces the likelihood of correct solutions for others, leading to the decline of Pass@k performance, or the probability of generating a correct solution within k attempts. Second, we uncover the winner-take-all phenomenon: RLVR disproportionately reinforces problems with high likelihood, correct solutions, under the base model, while suppressing other initially low-likelihood ones. Through extensive theoretical and empirical analysis on multiple mathematical reasoning benchmarks, we show that this effect arises from the inherent on-policy sampling in standard RL objectives, causing the model to converge toward narrow solution strategies. Based on these insights, we propose a simple yet effective data curation algorithm that focuses RLVR learning on low-likelihood problems, achieving notable improvement in Pass@k performance. Our code is available at https://github.com/mail-research/SELF-llm-interference.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 32 of its 46 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

32 of 46 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
  2. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning2 Jun
  3. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  4. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  5. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining10 Apr
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  9. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning10 Mar
  10. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  12. 2023Let's Verify Step by Step31 May
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 19

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  8. 2024Learning Dynamics of LLM Finetuningcited by 93
  9. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  10. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  11. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  12. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
7 more
  1. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  2. 2024A Study of Plasticity Loss in On-Policy Deep Reinforcement Learningcited by 26
  3. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  4. 2025Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churncited by 22
  5. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churncited by 16
  6. 2025Temporal Sampling for Forgotten Reasoning in LLMscited by 12
  7. 2020Towards a practical measure of interference for reinforcement learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 17.

17 papers.

Sort:

By area:

  1. 2026Maximum Likelihood Reinforcement Learningcited by 29
  2. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 13
  3. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 9
  4. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 5
  5. 2026Can RL Improve Generalization of LLM Agents? An Empirical Studycited by 4
  6. 2026Understanding and Mitigating Premature Confidence for Better LLM Reasoningcited by 4
  7. 2026CriPO: Enhancing Rubric-based RL via Self-Distillationcited by 4
  8. 2026GeoRA: Geometry-Aware Low-Rank Adaptation for RLVRcited by 3
  9. 2026The Unlearnability Phenomenon in RLVR for Language Modelscited by 2
  10. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVRcited by 1
  11. 2026Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewardscited by 1
  12. 2026Reachability Is Not Realization: Tracing the Sources of LLM Benchmark Gainscited by 1
5 more
  1. 2026OGER: A Robust Offline-Guided Exploration Reward for Hybrid Reinforcement Learningcited by 0
  2. 2026Reasoning Fine-Tuning Induces Persistent Latent Policy Statescited by 0
  3. 2026SLPO: Scaling Latent Reasoning via a Surrogate Policycited by 0
  4. 2026Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Spacecited by 0
  5. 2026On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
  3. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  4. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
  5. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
  6. 2025Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries
  7. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
  8. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation