# Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning

arXiv: https://arxiv.org/abs/2506.06632 · HTML: https://mlsanity.com/paper/2506.06632/

## Key facts

- Cited by 92 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 43 of its 53 references that are in the map.
- First posted on arXiv 7 Jun 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 43)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought](https://mlsanity.com/paper/2210.01240/) (2022)
- [On the Planning Abilities of Large Language Models : A Critical Investigation](https://mlsanity.com/paper/2305.15771/) (2023)
- [Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning](https://mlsanity.com/paper/2502.14768/) (2025)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Llama-Nemotron: Efficient Reasoning Models](https://mlsanity.com/paper/2505.00949/) (2025)
- [Stream of Search (SoS): Learning to Search in Language](https://mlsanity.com/paper/2404.03683/) (2024)

## Built on by (most cited first, 30 of 92)

- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation](https://mlsanity.com/paper/2507.13266/) (2025)
- [Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learning](https://mlsanity.com/paper/2508.03501/) (2025)
- [VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models](https://mlsanity.com/paper/2509.19803/) (2025)
- [RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning](https://mlsanity.com/paper/2510.02240/) (2025)
- [Imbalanced Gradients in RL Post-Training of Multi-Task LLMs](https://mlsanity.com/paper/2510.19178/) (2025)
- [Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2507.17512/) (2025)
- [NoRD: A Data-Efficient Vision-Language-Action Model that Drives without Reasoning](https://mlsanity.com/paper/2602.21172/) (2026)
- [BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning](https://mlsanity.com/paper/2510.26374/) (2025)
- [Multi-Task GRPO: Reliable LLM Reasoning Across Tasks](https://mlsanity.com/paper/2602.05547/) (2026)
- [Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability](https://mlsanity.com/paper/2601.18778/) (2026)
- [ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning](https://mlsanity.com/paper/2512.13095/) (2025)
- [Learning to Hint for Reinforcement Learning](https://mlsanity.com/paper/2604.00698/) (2026)
- [Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics](https://mlsanity.com/paper/2510.09901/) (2025)
- [What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoning](https://mlsanity.com/paper/2510.19099/) (2025)
- [VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL](https://mlsanity.com/paper/2511.18902/) (2025)
- [F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare](https://mlsanity.com/paper/2602.06717/) (2026)
- [Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation](https://mlsanity.com/paper/2606.06712/) (2026)
- [G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance](https://mlsanity.com/paper/2508.13023/) (2025)
- [h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning](https://mlsanity.com/paper/2510.07312/) (2025)
- [SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoning](https://mlsanity.com/paper/2601.04809/) (2026)
- [Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training](https://mlsanity.com/paper/2602.20532/) (2026)
- [EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning](https://mlsanity.com/paper/2508.07809/) (2025)
- [EvolProver: Advancing Automated Theorem Proving by Evolving Formalized Problems via Symmetry and Difficulty](https://mlsanity.com/paper/2510.00732/) (2025)
- [Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration](https://mlsanity.com/paper/2511.00794/) (2025)
- [Achieving Olympiad-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learning](https://mlsanity.com/paper/2512.10534/) (2025)
- [PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning](https://mlsanity.com/paper/2602.21992/) (2026)
- [RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling](https://mlsanity.com/paper/2506.08672/) (2025)
- [Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Data](https://mlsanity.com/paper/2601.15158/) (2026)

## Abstract (arXiv)

We aim to improve the reasoning capabilities of language models via reinforcement learning (RL). Recent RL post-trained models like DeepSeek-R1 have demonstrated reasoning abilities on mathematical and coding tasks. However, prior studies suggest that using RL alone to improve reasoning on inherently difficult tasks is less effective. Here, we draw inspiration from curriculum learning and propose to schedule tasks from easy to hard (E2H), allowing LLMs to build reasoning skills gradually. Our method is termed E2H Reasoner. Empirically, we observe that, although easy tasks are important initially, fading them out through appropriate scheduling is essential in preventing overfitting. Theoretically, we establish convergence guarantees for E2H Reasoner within an approximate policy iteration framework. We derive finite-sample complexity bounds and show that when tasks are appropriately decomposed and conditioned, learning through curriculum stages requires fewer total samples than direct learning. Experiments across multiple domains show that E2H Reasoner significantly improves the reasoning ability of small LLMs (1.5B to 3B), which otherwise struggle when trained with vanilla RL alone, highlighting the effectiveness of our method. Our code can be found on https://github.com/divelab/E2H-Reasoning.