mlsanityData to 14 Oct 2026

Self-Evolving Curriculum for LLM Reasoning

arXiv 2505.14970 ↗First posted 20 May 2025cs.AI cs.LGcited by 112

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Reinforcement learning (RL) has proven effective for fine-tuning large language models (LLMs), significantly enhancing their reasoning abilities in domains such as mathematics and code generation. A crucial factor influencing RL fine-tuning success is the training curriculum: the order in which training problems are presented. While random curricula serve as common baselines, they remain suboptimal; manually designed curricula often rely heavily on heuristics, and online filtering methods can be computationally prohibitive. To address these limitations, we propose Self-Evolving Curriculum (SEC), an automatic curriculum learning method that learns a curriculum policy concurrently with the RL fine-tuning process. Our approach formulates curriculum selection as a non-stationary Multi-Armed Bandit problem, treating each problem category (e.g., difficulty level or problem type) as an individual arm. We leverage the absolute advantage from policy gradient methods as a proxy measure for immediate learning gain. At each training step, the curriculum policy selects categories to maximize this reward signal and is updated using the TD(0) method. Across three distinct reasoning domains: planning, inductive reasoning, and mathematics, our experiments demonstrate that SEC significantly improves models' reasoning capabilities, enabling better generalization to harder, out-of-distribution test problems. Additionally, our approach achieves better skill balance when fine-tuning simultaneously on multiple reasoning domains. These findings highlight SEC as a promising strategy for RL fine-tuning of LLMs.

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 49 of its 69 references that are in the map. First posted on arXiv 20 May 2025.

Builds on

49 of 69 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training13 Apr
  2. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  3. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  4. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  7. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution25 Feb
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  9. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  11. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb
  12. 2023Let's Verify Step by Step31 May
3 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Solving math word problems with process- and outcome-based feedback25 Nov
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2015Prioritized Experience Replaycited by 1,099
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  12. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
22 more
  1. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  2. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 392
  3. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  4. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  5. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  6. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  7. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  8. 2017Teacher-Student Curriculum Learningcited by 210
  9. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 198
  10. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  11. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  12. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  13. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  14. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 150
  15. 2015Online Batch Selection for Faster Training of Neural Networkscited by 122
  16. 2023Amortizing intractable inference in large language modelscited by 97
  17. 2023LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representationscited by 82
  18. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  19. 2025Training a Generally Curious Agentcited by 31
  20. 2024Adaptive teachers for amortized samplerscited by 23
  21. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 20
  22. 2024Proof Flow: Preliminary Study on Generative Flow Network Language Model Tuning for Formal Reasoningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 83.

112 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  3. 2026AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolutioncited by 115
  4. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 92
  5. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 52
  6. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 50
  7. 2026Maximum Likelihood Reinforcement Learningcited by 36
  8. 2025Diversity-Incentivized Exploration for Versatile Reasoningcited by 28
  9. 2025Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocationcited by 22
  10. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 20
  11. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMscited by 19
  12. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learningcited by 17
38 more
  1. 2025RL for Reasoning by Adaptively Revealing Rationalescited by 16
  2. 2025BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuningcited by 14
  3. 2026Graph is a Substrate Across Data Modalitiescited by 14
  4. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 14
  5. 2026Multi-Task GRPO: Reliable LLM Reasoning Across Taskscited by 14
  6. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  7. 2026CoBA-RL: Capability-Oriented Budget Allocation for Reinforcement Learning in LLMscited by 12
  8. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  9. 2025What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoningcited by 11
  10. 2025VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RLcited by 11
  11. 2026Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Modelscited by 11
  12. 2025CEC-Zero: Zero-Supervision Character Error Correction with Self-Generated Rewardscited by 10
  13. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Modelcited by 10
  14. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 10
  15. 2026Understanding Diversity Collapse in RLVR via the Lens of Overtrainingcited by 10
  16. 2025DynamixSFT: Dynamic Mixture Optimization of Instruction Tuning Collectionscited by 9
  17. 2025LatentEvolve: Self-Evolving Test-Time Scaling in Latent Spacecited by 9
  18. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  19. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 9
  20. 2025Dynamic Sampling that Adapts: Self-Aware Iterative Data Persistent Optimization for Mathematical Reasoningcited by 8
  21. 2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 8
  22. 2026Active Zero: Self-Evolving Vision-Language Models through Active Environment Explorationcited by 8
  23. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 8
  24. 2026Buffer Matters: Unleashing the Power of Off-Policy Reinforcement Learning in Large Language Model Reasoningcited by 8
  25. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
  26. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learningcited by 7
  27. 2025What Can You Do When You Have Zero Rewards During RL?cited by 7
  28. 2025Efficient Reinforcement Learning for Large Language Models with Intrinsic Explorationcited by 7
  29. 2025Achieving Olympiad-Level Geometry Large Language Model Agent via Complexity Boosting Reinforcement Learningcited by 7
  30. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptationcited by 7
  31. 2026LongCat-Flash-Prover: Advancing Native Formal Reasoning via Agentic Tool-Integrated Reinforcement Learningcited by 7
  32. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  33. 2025Reinforcement Mid-Trainingcited by 6
  34. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedbackcited by 6
  35. 2026GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPOcited by 6
  36. 2026Autonomous Continual Learning for Environment Adaptation of Computer-Use Agentscited by 6
  37. 2026Weak-Driven Learning: How Weak Agents make Strong Agents Strongercited by 5
  38. 2026RuCL: Stratified Rubric-Based Curriculum Learning for Multimodal Large Language Model Reasoningcited by 5

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
  2. 2025Reasoning Curriculum: Bootstrapping Broad LLM Reasoning from Math
  3. 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
  4. 2026Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuning
  5. 2025Progressive Mastery: Customized Curriculum Learning with Guided Prompting for Mathematical Reasoning
  6. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation
  7. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Training
  8. 2025Learning to Reason at the Frontier of Learnability