mlsanityData to 14 Oct 2026

The Invisible Leash: Why RLVR May or May Not Escape Its Origin

arXiv 2507.14843 ↗First posted 20 Jul 2025cs.LG cs.AI cs.CLcited by 97

Not on the map. Why?

Cited by 54 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances highlight Reinforcement Learning with Verifiable Rewards (RLVR) as a promising approach to enhancing LLM capabilities. However, it remains unclear whether the current practice of RLVR truly expands models' reasoning boundary or mainly amplifies high-reward outputs that base models already know. We introduce empirical support dynamics, a framework that quantifies how RLVR preserves, expands, and shrinks access to verified solutions. Applied across models, reasoning tasks, and training checkpoints, it matches verified successes question by question to reveal changes that aggregate accuracy can conceal. Across the evaluated model-level benchmark aggregates, textitthe shrinkage of observed support outweighs its expansion: more questions lose a verified success than gain one at the stated sampling budgets. These losses occur alongside gains in textttpass@1. Some RLVR models increase average token-level entropy while reducing final-answer entropy: their token choices become more uncertain on average even as their final answers become more concentrated. Together, these findings reveal an ``invisible leash'' in current RLVR practice: it can make existing successes more reliable while losing access to other verified solutions. Empirical support dynamics make this imbalance measurable and motivate exploration strategies that reach underrepresented solutions while retaining prior successes.

Key facts. Cited by 97 papers in the map (data through 14 Oct 2026). Builds on 44 of its 62 references that are in the map. First posted on arXiv 20 Jul 2025.

Builds on

44 of 62 references.

References · 44

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
32 more
  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  2. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
  3. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 511
  4. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  5. 2024Measuring short-form factuality in large language modelscited by 373
  6. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  7. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Taskscited by 285
  8. 2023The Impact of AI on Developer Productivity: Evidence from GitHub Copilotcited by 274
  9. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  10. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 256
  11. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 216
  12. 2025Spurious Rewards: Rethinking Training Signals in RLVRcited by 209
  13. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  14. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  15. 2025Phi-4-reasoning Technical Reportcited by 151
  16. 2025Med-R1: Reinforcement Learning for Generalizable Medical Reasoning in Vision-Language Modelscited by 144
  17. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  18. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretrainingcited by 129
  19. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
  20. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questionscited by 97
  21. 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learningcited by 87
  22. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 80
  23. 2024Are Your LLMs Capable of Stable Reasoning?cited by 65
  24. 2025Rethinking Reflection in Pre-Trainingcited by 43
  25. 2025Med-RLVR: Emerging Medical Reasoning from a 3B base model via reinforcement Learningcited by 40
  26. 2025Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewardscited by 38
  27. 2025Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoningcited by 34
  28. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  29. 2025Real-World Doctor Agent with Proactive Consultation through Multi-Agent Reinforcement Learningcited by 27
  30. 2023Online Advertisements with LLMs: Opportunities and Challengescited by 17
  31. 2025Temporal Sampling for Forgotten Reasoning in LLMscited by 12
  32. 2024A Framework for Collaborating a Large Language Model Tool in Brainstorming for Triggering Creative Thoughtscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 84.

97 papers.

Sort:

By area:

  1. 2026Self-Distillation Enables Continual Learningcited by 247
  2. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 69
  3. 2025Understanding Tool-Integrated Reasoningcited by 38
  4. 2026Maximum Likelihood Reinforcement Learningcited by 36
  5. 2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34
  6. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 22
  7. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoningcited by 21
  8. 2026On the Geometry of On-Policy Distillationcited by 19
  9. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  10. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  11. 2026Learn Hard Problems During RL with Reference Guided Fine-tuningcited by 12
  12. 2026AI Can Learn Scientific Tastecited by 11
38 more
  1. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 10
  2. 2026On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversitycited by 10
  3. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learningcited by 9
  4. 2026DSDR: Dual-Scale Diversity Regularization for Exploration in LLM Reasoningcited by 9
  5. 2026Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversitycited by 9
  6. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoningcited by 6
  7. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 6
  8. 2026Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewardscited by 6
  9. 2026Large Language Model Post-Training: A Unified View of Off-Policy and On-Policy Learningcited by 6
  10. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?cited by 5
  11. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  12. 2026Poly-EPO: Training Exploratory Reasoning Modelscited by 5
  13. 2026Where to Spend Rollouts: Hit-Utility Optimal Rollout Allocation for Group-Based RLVRcited by 5
  14. 2026Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillationcited by 5
  15. 2026Towards Understanding On-Policy Distillation through the Lens of Test-Time Scalingcited by 5
  16. 2025Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMscited by 4
  17. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 4
  18. 2026Linear Dynamics in the RLVR Training of Large Language Modelscited by 4
  19. 2026GeoRA: Geometry-Aware Low-Rank Adaptation for RLVRcited by 4
  20. 2026A Relative-Budget Theory for Reinforcement Learning with Verifiable Rewards in Large Language Model Reasoningcited by 4
  21. 2026Monitorability as a Free Gift: How RLVR Spontaneously Aligns Reasoningcited by 4
  22. 2026Text Before Vision: Staged Knowledge Injection Matters for Agentic RLVR in Ultra-High-Resolution Remote Sensing Understandingcited by 4
  23. 2026The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimationcited by 4
  24. 2026Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problemscited by 4
  25. 2026On the Generalization Gap in Self-Evolving Language Model Reasoningcited by 4
  26. 2025Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversitycited by 3
  27. 2026Dr. Seg: Revisiting GRPO Training for Visual Large Language Models through Perception-Oriented Designcited by 3
  28. 2026The Unlearnability Phenomenon in RLVR for Language Modelscited by 3
  29. 2026Tail-Likelihood Reinforcement Learningcited by 3
  30. 2026RL Starts before RL: On Policy Distillation for Better Reinforcement Learningcited by 3
  31. 2026Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMscited by 2
  32. 2026Policy Improvement Reinforcement Learningcited by 2
  33. 2026How You Begin is How You Reason: Driving Exploration in RLVR via Prefix-Tuned Priorscited by 2
  34. 2026AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignmentcited by 2
  35. 2026Zipping the Thought: When and How Compressed Reasoning Data Works in LLM Post-Trainingcited by 2
  36. 2026Where Rollouts Begin: Low-Load, High-Leverage First-Token Diversification for RLVRcited by 2
  37. 2026Extrapolative Weight Averaging Reveals Correctness-Efficiency Frontiers in Code RLcited by 2
  38. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2

50 most cited of 97 All 97 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation
  2. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  3. 2026Detecting RLVR Training Data via Structural Convergence of Reasoning
  4. 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
  5. 2026Rate or Fate? RLV^varepsilonR: Reinforcement Learning with Verifiable Noisy Rewards
  6. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  7. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs
  8. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation