mlsanityData to 14 Oct 2026

VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models

arXiv 2509.19803 ↗First posted 24 Sep 2025cs.LG cs.CLcited by 36

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 50 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

43 of 50 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning7 Jun
  2. 2025Learning to Reason under Off-Policy Guidance21 Apr
  3. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning18 Apr
  4. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification9 Mar
  9. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  10. 2025Ignore the KL Penalty! Boosting Exploration on Critical Tokens to Enhance RL Fine-Tuning10 Feb
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
3 more
  1. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb
  2. 2023Let's Verify Step by Step31 May
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 28

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  7. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  8. 2025Demystifying Long Chain-of-Thought Reasoning in LLMscited by 326
  9. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  10. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 293
  11. 2021Curriculum Learning: A Surveycited by 259
  12. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 223
16 more
  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  3. 2025LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!cited by 132
  4. 2024Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capabilitycited by 83
  5. 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learningcited by 71
  6. 2024LLM With Tools: A Surveycited by 61
  7. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  8. 2025Code to Think, Think to Code: A Survey on Code-Enhanced Reasoning and Reasoning-Driven Code Intelligence in LLMscited by 47
  9. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  10. 2025FlashThink: An Early Exit Method For Efficient Reasoningcited by 28
  11. 2025Train Long, Think Short: Curriculum Learning for Efficient Reasoningcited by 20
  12. 2025DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learningcited by 14
  13. 2024ToNER: Type-oriented Named Entity Recognition with Generative Language Modelcited by 8
  14. 2025PVPO: Pre-Estimated Value-Based Policy Optimization for Agentic Reasoningcited by 4
  15. 2024Mitigating Out-of-Entity Errors in Named Entity Recognition: A Sentence-Level Strategycited by 3
  16. 2025Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understandingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 32.

36 papers.

Sort:

By area:

  1. 2026HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillationcited by 42
  2. 2025VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RLcited by 11
  3. 2026VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reductioncited by 10
  4. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  5. 2026Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Informationcited by 7
  6. 2026GOPO: Policy Optimization using Ranked Rewardscited by 5
  7. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  8. 2026Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVRcited by 5
  9. 2025PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMscited by 4
  10. 2026Learning Agentic Policy from Action Guidancecited by 4
  11. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learningcited by 3
  12. 2026sGPO: Trading Inference FLOPs for Training Efficiency in RLVRcited by 3
24 more
  1. 2026Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learningcited by 3
  2. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  3. 2025Accelerating Inference of Masked Image Generators via Reinforcement Learningcited by 1
  4. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspectivecited by 1
  5. 2026Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMscited by 1
  6. 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1
  7. 2026Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carriercited by 1
  8. 2026CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewardscited by 1
  9. 2025Studying the Korean Word-Chain Game with RLVR: Mitigating Reward Conflicts via Curriculum Learningcited by 0
  10. 2026Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Modelscited by 0
  11. 2026Level Up: Defining and Exploiting Transitional Problems for Curriculum Learningcited by 0
  12. 2026Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuningcited by 0
  13. 2026What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QAcited by 0
  14. 2026Reasoning Depth and Environment Complexity: A Controlled Study of RLVR Data Allocation across Logical Reasoning Taskscited by 0
  15. 2026Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Modelscited by 0
  16. 2026LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RLcited by 0
  17. 2026From Rollouts to Recipes: Self-Contained Post-Training for LLMscited by 0
  18. 2026Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoningcited by 0
  19. 2026CHART: A Harness-Rotation Curriculum for Harness-Robust Search Agentscited by 0
  20. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  21. 2026RLVR landscapes for iterated multiplications can be benign: Insights from spin-glass theorycited by 0
  22. 2026Selecting Diverse SFT Traces Improves Post-RL Generalizationcited by 0
  23. 2026Prompts Live on an Arc: Gaussian Curricula in Fisher--Rao Coordinates for Rollout-Efficient GRPOcited by 0
  24. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning
  2. 2025Self-Evolving Curriculum for LLM Reasoning
  3. 2025Reasoning Curriculum: Bootstrapping Broad LLM Reasoning from Math
  4. 2026From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning
  5. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning
  6. 2025CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
  7. 2026Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards
  8. 2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model