mlsanityData to 14 Oct 2026

Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning

arXiv 2506.06632 ↗First posted 7 Jun 2025cs.LG cs.AI cs.CLcited by 92

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We aim to improve the reasoning capabilities of language models via reinforcement learning (RL). Recent RL post-trained models like DeepSeek-R1 have demonstrated reasoning abilities on mathematical and coding tasks. However, prior studies suggest that using RL alone to improve reasoning on inherently difficult tasks is less effective. Here, we draw inspiration from curriculum learning and propose to schedule tasks from easy to hard (E2H), allowing LLMs to build reasoning skills gradually. Our method is termed E2H Reasoner. Empirically, we observe that, although easy tasks are important initially, fading them out through appropriate scheduling is essential in preventing overfitting. Theoretically, we establish convergence guarantees for E2H Reasoner within an approximate policy iteration framework. We derive finite-sample complexity bounds and show that when tasks are appropriately decomposed and conditioned, learning through curriculum stages requires fewer total samples than direct learning. Experiments across multiple domains show that E2H Reasoner significantly improves the reasoning ability of small LLMs (1.5B to 3B), which otherwise struggle when trained with vanilla RL alone, highlighting the effectiveness of our method. Our code can be found on https://github.com/divelab/E2H-Reasoning.

Key facts. Cited by 92 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 53 references that are in the map. First posted on arXiv 7 Jun 2025.

Builds on

43 of 53 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Self-Evolving Curriculum for LLM Reasoning20 May
  2. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  3. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  4. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  7. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  8. 2025Learning to Reason at the Frontier of Learnability17 Feb
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2024Qwen2.5 Technical Reportcited by 4,942
  8. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  9. 2019Invariant Risk Minimizationcited by 1,854
  10. 2025s1: Simple test-time scalingcited by 1,341
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  12. 2020In Search of Lost Domain Generalizationcited by 1,054
20 more
  1. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  2. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  3. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  4. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
  5. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  6. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 337
  7. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  8. 2017Automated Curriculum Learning for Neural Networkscited by 257
  9. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
  10. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  11. 2024Chain of Thoughtlessness? An Analysis of CoT in Planningcited by 103
  12. 2024LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBenchcited by 94
  13. 2024LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Modelscited by 70
  14. 2023Natural Language Reasoning, A Surveycited by 61
  15. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  16. 2014Approximate Policy Iteration Schemes: A Comparisoncited by 39
  17. 2025Training a Generally Curious Agentcited by 31
  18. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insightscited by 30
  19. 2025Complex LLM Planning via Automated Heuristics Discoverycited by 6
  20. 2022An Approximate Policy Iteration Viewpoint of Actor-Critic Algorithmscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 32; 2026: 60.

92 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  3. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 44
  4. 2025Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learningcited by 36
  5. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Modelscited by 36
  6. 2025RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learningcited by 21
  7. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 20
  8. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learningcited by 17
  9. 2026NoRD: A Data-Efficient Vision-Language-Action Model that Drives without Reasoningcited by 15
  10. 2025BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuningcited by 14
  11. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 14
  12. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
38 more
  1. 2025ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learningcited by 12
  2. 2026Learning to Hint for Reinforcement Learningcited by 12
  3. 2025Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physicscited by 11
  4. 2025What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoningcited by 11
  5. 2025VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RLcited by 11
  6. 2026F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rarecited by 11
  7. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  8. 2025G2RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidancecited by 10
  9. 2025h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learningcited by 9
  10. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 8
  11. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 8
  12. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learningcited by 7
  13. 2025EvolProver: Advancing Automated Theorem Proving by Evolving Formalized Problems via Symmetry and Difficultycited by 7
  14. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Explorationcited by 7
  15. 2025Achieving Olympiad-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learningcited by 7
  16. 2026PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learningcited by 7
  17. 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Samplingcited by 6
  18. 2026Outcome-Based RL Provably Leads Transformers to Reason, but Only With the Right Datacited by 6
  19. 2026Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learningcited by 6
  20. 2026Efficient RLVR Training via Weighted Mutual Information Data Selectioncited by 6
  21. 2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 6
  22. 2026RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoningcited by 5
  23. 2026Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVRcited by 5
  24. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  25. 2025Sample-efficient LLM Optimization with Reset Replaycited by 4
  26. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 4
  27. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  28. 2026Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoningcited by 4
  29. 2026Cog-DRIFT: Exploration on Adaptively Reformulated Instances Enables Learning from Hard Reasoning Problemscited by 4
  30. 2026SEAD: Competence-Aware On-Policy Distillation via Entropy-Guided Supervisioncited by 4
  31. 2025SA-GCS: Semantic-Aware Gaussian Curriculum Scheduling for UAV Vision-Language Navigationcited by 3
  32. 2025STaR: Towards Effective and Stable Table Reasoning via Slow-Thinking Large Language Modelscited by 3
  33. 2025RL-Struct: A Lightweight Reinforcement Learning Framework for Reliable Structured Output in LLMscited by 3
  34. 2026ThinkDrive: Chain-of-Thought Guided Progressive Reinforcement Learning Fine-Tuning for Autonomous Drivingcited by 3
  35. 2026PhysProver: Advancing Automatic Theorem Proving for Physicscited by 3
  36. 2026DeReason: A Difficulty-Aware Curriculum Improves Decoupled SFT-then-RL Training for General Reasoningcited by 3
  37. 2026DRTriton: Large-Scale Synthetic Data Driven Reinforcement Learning for Triton Kernel Generationcited by 3
  38. 2026Learnability-Informed Fine-Tuning of Diffusion Language Modelscited by 3

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Curriculum: Bootstrapping Broad LLM Reasoning from Math
  2. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective
  3. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs
  4. 2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model
  5. 2025Code-enabled language models can outperform reasoning models on diverse tasks
  6. 2025SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning
  7. 2025Incentivizing Strong Reasoning from Weak Supervision
  8. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning