# The Invisible Leash: Why RLVR May or May Not Escape Its Origin

arXiv: https://arxiv.org/abs/2507.14843 · HTML: https://mlsanity.com/paper/2507.14843/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 62 references that are in the map.
- First posted on arXiv 20 Jul 2025.

## Builds on (most cited first, 30 of 44)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs](https://mlsanity.com/paper/2506.14245/) (2025)
- [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [LiveBench: A Challenging, Contamination-Limited LLM Benchmark](https://mlsanity.com/paper/2406.19314/) (2024)
- [SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models](https://mlsanity.com/paper/2307.10635/) (2023)
- [AutoAgents: A Framework for Automatic Agent Generation](https://mlsanity.com/paper/2309.17288/) (2023)
- [Spurious Rewards: Rethinking Training Signals in RLVR](https://mlsanity.com/paper/2506.10947/) (2025)
- [ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models](https://mlsanity.com/paper/2505.24864/) (2025)
- [The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning](https://mlsanity.com/paper/2506.01347/) (2025)
- [Phi-4-reasoning Technical Report](https://mlsanity.com/paper/2504.21318/) (2025)
- [Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Models](https://mlsanity.com/paper/2503.13939/) (2025)
- [REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2505.24760/) (2025)
- [Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining](https://mlsanity.com/paper/2504.07912/) (2025)

## Built on by (most cited first, 30 of 97)

- [Self-Distillation Enables Continual Learning](https://mlsanity.com/paper/2601.19897/) (2026)
- [On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models](https://mlsanity.com/paper/2512.07783/) (2025)
- [Understanding Tool-Integrated Reasoning](https://mlsanity.com/paper/2508.19201/) (2025)
- [Maximum Likelihood Reinforcement Learning](https://mlsanity.com/paper/2602.02710/) (2026)
- [RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs](https://mlsanity.com/paper/2509.21128/) (2025)
- [Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning](https://mlsanity.com/paper/2511.19942/) (2025)
- [Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning](https://mlsanity.com/paper/2511.12344/) (2025)
- [On the Geometry of On-Policy Distillation](https://mlsanity.com/paper/2606.07082/) (2026)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2509.21882/) (2025)
- [Learn Hard Problems During RL with Reference Guided Fine-tuning](https://mlsanity.com/paper/2603.01223/) (2026)
- [AI Can Learn Scientific Taste](https://mlsanity.com/paper/2603.14473/) (2026)
- [Multi-Rollout On-Policy Distillation via Peer Successes and Failures](https://mlsanity.com/paper/2605.12652/) (2026)
- [On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity](https://mlsanity.com/paper/2606.26091/) (2026)
- [h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning](https://mlsanity.com/paper/2510.07312/) (2025)
- [DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoning](https://mlsanity.com/paper/2602.19895/) (2026)
- [Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity](https://mlsanity.com/paper/2605.00365/) (2026)
- [Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning](https://mlsanity.com/paper/2510.27044/) (2025)
- [Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures](https://mlsanity.com/paper/2601.19928/) (2026)
- [Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2602.02555/) (2026)
- [Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learning](https://mlsanity.com/paper/2604.07941/) (2026)
- [Can GRPO Help LLMs Transcend Their Pretraining Origin?](https://mlsanity.com/paper/2510.15990/) (2025)
- [Aletheia: What Makes RLVR For Code Verifiers Tick?](https://mlsanity.com/paper/2601.12186/) (2026)
- [Poly-EPO: Training Exploratory Reasoning Models](https://mlsanity.com/paper/2604.17654/) (2026)
- [Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVR](https://mlsanity.com/paper/2605.07114/) (2026)
- [Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillation](https://mlsanity.com/paper/2608.09826/) (2026)
- [Towards Understanding On-Policy Distillation through the Lens of Test-Time Scaling](https://mlsanity.com/paper/2608.11829/) (2026)
- [Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs](https://mlsanity.com/paper/2511.05933/) (2025)
- [Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectories](https://mlsanity.com/paper/2511.07368/) (2025)
- [Linear Dynamics in the RLVR Training of Large Language Models](https://mlsanity.com/paper/2601.04537/) (2026)

## Abstract (arXiv)

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, textitthe shrinkage of observed support outweighs its expansion: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in textttpass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.