mlsanityData to 14 Oct 2026

SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning

arXiv 2506.09016 ↗First posted 10 Jun 2025cs.LGcited by 29

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Training large language models with reinforcement learning (RL) against verifiable rewards significantly enhances their reasoning abilities, yet remains computationally expensive due to inefficient uniform prompt sampling. We introduce Selective Prompting with Efficient Estimation of Difficulty (SPEED), an adaptive online RL curriculum that selectively chooses training examples of intermediate difficulty to maximize learning efficiency. Theoretically, we establish that intermediate-difficulty prompts improve the gradient estimator's signal-to-noise ratio, accelerating convergence. Empirically, our efficient implementation leads to 2x to 6x faster training without degrading accuracy, requires no manual tuning, and integrates seamlessly into standard RL algorithms.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 59 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

44 of 59 references.

Closest ancestors · 17

Same direction, newest first

  1. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  2. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  3. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  4. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  5. 2025CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models28 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  8. 2025Learning to Reason at the Frontier of Learnability17 Feb
  9. 2025LIMR: Less is More for RL Scaling17 Feb
  10. 2025Training Language Models to Reason Efficiently6 Feb
  11. 2025Process Reinforcement through Implicit Rewards3 Feb
  12. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
5 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  2. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2025s1: Simple test-time scalingcited by 1,341
  8. 2024OpenAI o1 System Cardcited by 1,244
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2024DeepSeek-V3 Technical Reportcited by 811
  11. 2018Rethinking the Value of Network Pruningcited by 599
  12. 2025LIMO: Less is More for Reasoningcited by 498
15 more
  1. 2025MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learningcited by 303
  2. 2024Token-Budget-Aware LLM Reasoningcited by 247
  3. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 235
  4. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  5. 2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
  6. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  7. 2023Online Speculative Decodingcited by 113
  8. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  9. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 109
  10. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
  11. 2025Concise Reasoning via Reinforcement Learningcited by 79
  12. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  13. 2025Training a Generally Curious Agentcited by 31
  14. 2024Curriculum Learning for Small Code Language Modelscited by 22
  15. 2023Instruction Tuning with Human Curriculumcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 18.

29 papers.

Sort:

By area:

  1. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  2. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 35
  3. 2025It Takes Two: Your GRPO Is Secretly DPOcited by 26
  4. 2025Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectivescited by 25
  5. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 20
  6. 2025CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMscited by 19
  7. 2026Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewardscited by 17
  8. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 14
  9. 2026Train Less, Learn More: Adaptive Efficient Rollout Optimization for Group-Based Reinforcement Learningcited by 14
  10. 2025What Makes a Good Curriculum? Disentangling the Effects of Data Ordering on LLM Mathematical Reasoningcited by 11
  11. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 8
  12. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 8
17 more
  1. 2026MC-GRPO: Median-Centered Group Relative Policy Optimization for Small-Rollout Reinforcement Learningcited by 6
  2. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  3. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholdscited by 4
  4. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 4
  5. 2025Provable Benefit of Curriculum in Transformer Tree-Reasoning Post-Trainingcited by 4
  6. 2026HeaPA: Difficulty-Aware Heap Sampling and On-Policy Query Augmentation for LLM Reinforcement Learningcited by 4
  7. 2026DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimationcited by 2
  8. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  9. 2026Cross-Epoch Adaptive Rollout Optimization for RL Post-Trainingcited by 2
  10. 2026LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Modelscited by 2
  11. 2025From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Taskscited by 0
  12. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutionscited by 0
  13. 2026Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuningcited by 0
  14. 2026Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Modelscited by 0
  15. 2026Dynamic Important Example Mining for Reinforcement Finetuningcited by 0
  16. 2026MaPP: A Unified Marginalized Posterior-Predictive Framework for Data-Efficient RLVRcited by 0
  17. 2026CERO: Where and When to Allocate Rollouts for RL Post-Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  2. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
  3. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning
  4. 2026Train at Moving Edge: Online-Verified Prompt Selection for Efficient RL Training of Large Reasoning Model
  5. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?
  6. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  7. 2025Self-Evolving Curriculum for LLM Reasoning
  8. 2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model