# You Need Reasoning to Learn Reasoning: The Limitations of Label-Free RL in Weak Base Models

arXiv: https://arxiv.org/abs/2511.04902 · HTML: https://mlsanity.com/paper/2511.04902/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 7 Nov 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [TTRL: Test-Time Reinforcement Learning](https://mlsanity.com/paper/2504.16084/) (2025)
- [Learning to Reason without External Rewards](https://mlsanity.com/paper/2505.19590/) (2025)
- [GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism](https://mlsanity.com/paper/2501.07890/) (2025)
- [Towards NWDAF-enabled Analytics and Closed-Loop Automation in 5G Networks](https://mlsanity.com/paper/2505.06789/) (2025)
- [Protocol Learning, Decentralized Frontier Risk and the No-Off Problem](https://mlsanity.com/paper/2412.07890/) (2024)
- [First Field-Trial Demonstration of L4 Autonomous Optical Network for Distributed AI Training Communication: An LLM-Powered Multi-AI-Agent Solution](https://mlsanity.com/paper/2504.01234/) (2025)
- [Relative Optimal Transport](https://mlsanity.com/paper/2411.05678/) (2024)

## Built on by (most cited first, 2 of 2)

- [MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Training](https://mlsanity.com/paper/2608.10823/) (2026)
- [Group-Marginalized Self-Rewarding RL Drives Zero-Label Self-Evolving](https://mlsanity.com/paper/2609.36750/) (2026)

## Abstract (arXiv)

Recent advances in large language models have demonstrated the promise of unsupervised reinforcement learning (RL) methods for enhancing reasoning capabilities without external supervision. However, the generalizability of these label-free RL approaches to smaller base models with limited reasoning capabilities remains unexplored. In this work, we systematically investigate the performance of label-free RL methods across different model sizes and reasoning strengths, from 0.5B to 7B parameters. Our empirical analysis reveals critical limitations: label-free RL is highly dependent on the base model's pre-existing reasoning capability, with performance often degrading below baseline levels for weaker models. We find that smaller models fail to generate sufficiently long or diverse chain-of-thought reasoning to enable effective self-reflection, and that training data difficulty plays a crucial role in determining success. To address these challenges, we propose a simple yet effective method for label-free RL that utilizes curriculum learning to progressively introduce harder problems during training and mask no-majority rollouts during training. Additionally, we introduce a data curation pipeline to generate samples with predefined difficulty. Our approach demonstrates consistent improvements across all model sizes and reasoning capabilities, providing a path toward more robust unsupervised RL that can bootstrap reasoning abilities in resource-constrained models. We make our code available at https://github.com/BorealisAI/CuMa