mlsanityData to 7 Oct 2026

Prompt Curriculum Learning for Efficient LLM Post-Training

arXiv 2510.01135 ↗First posted 1 Oct 2025cs.LG cs.CLcited by 40

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce Prompt Curriculum Learning (PCL), a lightweight reinforcement learning (RL) algorithm that selects intermediate-difficulty prompts using a learned value model to post-train language models. Since post-training LLMs via RL remains sensitive to batching and prompt selection strategies, we first conduct a series of systematic experiments where we (1) determine the optimal training batch size that balances generation efficiency and gradient quality and (2) establish the importance of focusing on prompts of intermediate difficulty for the policy. We build upon these results to design PCL, which identifies prompts of intermediate difficulty for the current policy in an on-policy manner by using a value model that is concurrently updated based on the current policy. By focusing on informative prompts that yield high effective ratios, PCL achieves either the highest performance or requires significantly less time to reach comparable performance to its counterparts. Compared to rollout-based filtering methods, PCL avoids costly rollouts and achieves 12.1× and 16.9× faster speed on identifying intermediate-difficulty prompts when training on MATH and DeepScaleR, respectively. We further demonstrate that our value model accurately predicts prompt difficulty and allows PCL to focus on progressively more challenging prompts during RL. Our results present a new methodology that delivers improved tradeoff between upper-bound performance and efficiency for reasoning-focused RL.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

45 of 59 references.

References · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2025s1: Simple test-time scalingcited by 1,311
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
33 more
  1. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  2. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  3. 2019Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Boundscited by 602
  4. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  5. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  6. 2025LIMO: Less is More for Reasoningcited by 488
  7. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
  8. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisonscited by 244
  9. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  10. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
  11. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 168
  12. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  13. 2025A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforcecited by 141
  14. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  15. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learningcited by 88
  16. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  17. 2025An Empirical Study on Eliciting and Improving R1-like Reasoning Modelscited by 78
  18. 2020VarGrad: A Low-Variance Gradient Estimator for Variational Inferencecited by 73
  19. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  20. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  21. 2024Active Preference Learning for Large Language Modelscited by 55
  22. 2024Offline Reinforcement Learning for LLM Multi-Step Reasoningcited by 53
  23. 2025Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replaycited by 52
  24. 2019Ordered SGD: A New Stochastic Optimization Framework for Empirical Risk Minimizationcited by 51
  25. 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?cited by 48
  26. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  27. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  28. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regressioncited by 23
  29. 2024Enhancing Multi-Step Reasoning Abilities of Language Models through Direct Q-Function Optimizationcited by 22
  30. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  31. 2025Model Predictive Task Sampling for Efficient and Robust Adaptationcited by 20
  32. 2023Instruction Tuning with Human Curriculumcited by 15
  33. 2025Infinite Sampling: Efficient and Stable Grouped RL Training for Large Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 36.

40 papers.

Sort:

By area:

  1. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  2. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 43
  3. 2025Scaling Agent Learning via Experience Synthesiscited by 40
  4. 2025Imbalanced Gradients in RL Post-Training of Multi-Task LLMscited by 19
  5. 2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
  6. 2026Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoningcited by 10
  7. 2026TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learningcited by 9
  8. 2026Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Trainingcited by 7
  9. 2026Prompt replay: speeding up grpo with on-policy reuse of high-signal promptscited by 7
  10. 2026RLVR without Ineffective Samples: Group Prioritized Off-Policy Optimization for LLM Reasoningcited by 7
  11. 2026GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPOcited by 6
  12. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
28 more
  1. 2026Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioningcited by 3
  2. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  3. 2026Max Out GRPO Signal: Adaptive Trace Prefix Control for Hard Reasoning Problemscited by 3
  4. 2026Goldilocks RL: Tuning Task Difficulty to Escape Sparse Rewards for Reasoningcited by 2
  5. 2026Learning to Reason with Curriculum I: Provable Benefits of Autocurriculumcited by 2
  6. 2026Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplexcited by 2
  7. 2026Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal Statescited by 2
  8. 2026The Unlearnability Phenomenon in RLVR for Language Modelscited by 2
  9. 2026LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Modelscited by 2
  10. 2026p1: Better Prompt Optimization with Fewer Promptscited by 1
  11. 2026Guiding LLM Post-training Data Engineering with Model Internals from Sparse Autoencoderscited by 1
  12. 2026IRDS: Interpretable RLVR Data Selection via Verifier-Coupled Sparse Autoencoder Coveragecited by 1
  13. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 1
  14. 2026Beyond Solvability: Task Learnability as a Static Prior for LLM RL Post-Trainingcited by 1
  15. 2026ThinkPrior: Zero-Rollout Difficulty Priors for Cold-Start Prompt Selection in RLVRcited by 1
  16. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  17. 2026Limits of Difficulty Scaling: Hard Samples Yield Diminishing Returns in GRPO-Tuned SLMscited by 0
  18. 2026Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuningcited by 0
  19. 2026Curriculum Group Policy Optimization: Adaptive Sampling for Unleashing the Potential of Text-to-Image Generationcited by 0
  20. 2026SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learningcited by 0
  21. 2026SDO: Structure-Aware Data Organization for Efficient LLM Post-Trainingcited by 0
  22. 2026Dynamic Important Example Mining for Reinforcement Finetuningcited by 0
  23. 2026From Self-Distillation to Self-Practice: Privileged Information for Multi-Turn Agentscited by 0
  24. 2026Program-Verified Self-Evolution for Vision-Language Modelscited by 0
  25. 2026Towards Better Training Signal: Advantage Clipped Policy Optimizationcited by 0
  26. 2026Prompts Live on an Arc: Gaussian Curricula in Fisher--Rao Coordinates for Rollout-Efficient GRPOcited by 0
  27. 2026Unlearnable, or Unmeasured? On the Reliability of Difficulty Labels in RLVRcited by 0
  28. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillationcited by 0