# How Far Can Unsupervised RLVR Scale LLM Training?

arXiv: https://arxiv.org/abs/2603.08660 · HTML: https://mlsanity.com/paper/2603.08660/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Mar 2026.

## Built on by (most cited first, 30 of 30)

- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Embarrassingly Simple Self-Distillation Improves Code Generation](https://mlsanity.com/paper/2604.01193/) (2026)
- [Prune-OPD: Efficient and Reliable On-Policy Distillation for Long-Horizon Reasoning](https://mlsanity.com/paper/2605.07804/) (2026)
- [Rethinking On-Policy Distillation of Large Language Models II: One Training Example](https://mlsanity.com/paper/2609.04172/) (2026)
- [Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation](https://mlsanity.com/paper/2605.11739/) (2026)
- [Anti-Self-Distillation for Reasoning RL via Pointwise Mutual Information](https://mlsanity.com/paper/2605.11609/) (2026)
- [OPRD: On-Policy Representation Distillation](https://mlsanity.com/paper/2606.06021/) (2026)
- [When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2605.25864/) (2026)
- [EVE: Verifiable Self-Evolution of MLLMs via Executable Visual Transformations](https://mlsanity.com/paper/2604.18320/) (2026)
- [CreativityBench: Evaluating Agent Creative Reasoning via Affordance-Based Tool Repurposing](https://mlsanity.com/paper/2605.02910/) (2026)
- [Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL](https://mlsanity.com/paper/2605.22217/) (2026)
- [From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space](https://mlsanity.com/paper/2604.14142/) (2026)
- [Cross-lingual Self-Consistency for Multilingual Reasoning with Language Models](https://mlsanity.com/paper/2606.01464/) (2026)
- [PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching](https://mlsanity.com/paper/2603.18363/) (2026)
- [Policy Improvement Reinforcement Learning](https://mlsanity.com/paper/2604.00860/) (2026)
- [When Can LLMs Learn to Reason with Weak Supervision?](https://mlsanity.com/paper/2604.18574/) (2026)
- [RemoteZero: Geospatial Reasoning with Zero Labels](https://mlsanity.com/paper/2605.04451/) (2026)
- [Self-Evolving Deep Research via Joint Generation and Evaluation](https://mlsanity.com/paper/2606.04507/) (2026)
- [Continual Self-Improvement with Lightweight Experiential Latent Memories](https://mlsanity.com/paper/2606.17803/) (2026)
- [Neuron-Aware Data Selection for Annotation-Free LLM Self-Distillation](https://mlsanity.com/paper/2607.02460/) (2026)
- [A Theory of LLM Information Susceptibility](https://mlsanity.com/paper/2603.23626/) (2026)
- [Less Data Approximates More: Earning Faithful Confidence in High-Stakes Domains](https://mlsanity.com/paper/2604.08454/) (2026)
- [Two is better than one: A Collapse-free Multi-Reward RLIF Training Framework](https://mlsanity.com/paper/2605.22620/) (2026)
- [Be Your Own Teacher: Steering Protein Language Models via Unsupervised Reward Optimization](https://mlsanity.com/paper/2606.18961/) (2026)
- [AdaStop: Cost-Aware Early Stopping for DNN Test Selection](https://mlsanity.com/paper/2607.05461/) (2026)
- [ManGo: Manga Active Narrative Grounding Optimization](https://mlsanity.com/paper/2608.29865/) (2026)
- [Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation](https://mlsanity.com/paper/2609.09135/) (2026)
- [Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving](https://mlsanity.com/paper/2609.36750/) (2026)
- [Self-Evolving Visual Agent via Margin Reward](https://mlsanity.com/paper/2610.15084/) (2026)
- [Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVR](https://mlsanity.com/paper/2610.17824/) (2026)

## Abstract (arXiv)

Unsupervised reinforcement learning with verifiable rewards (URLVR) offers a pathway to scale LLM training beyond the supervision bottleneck by deriving rewards without ground truth labels. Recent works leverage model intrinsic signals, showing promising early gains, yet their potential and limitations remain unclear. In this work, we revisit URLVR and provide a comprehensive analysis spanning taxonomy, theory and extensive experiments. We first classify URLVR methods into intrinsic versus external based on reward sources, then establish a unified theoretical framework revealing that all intrinsic methods converge toward sharpening the model's initial distribution This sharpening mechanism succeeds when initial confidence aligns with correctness but fails catastrophically when misaligned. Through systematic experiments, we show intrinsic rewards consistently follow a rise-then-fall pattern across methods, with collapse timing determined by model prior rather than engineering choices. Despite these scaling limits, we find intrinsic rewards remain valuable in test-time training on small datasets, and propose Model Collapse Step to measure model prior, serving as a practical indicator for RL trainability. Finally, we explore external reward methods that ground verification in computational asymmetries, showing preliminary evidence they may escape the confidence-correctness ceiling. Our findings chart boundaries for intrinsic URLVR while motivating paths toward scalable alternatives.