# VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models

arXiv: https://arxiv.org/abs/2509.19803 · HTML: https://mlsanity.com/paper/2509.19803/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 43 of its 50 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 43)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [Understanding R1-Zero-Like Training: A Critical Perspective](https://mlsanity.com/paper/2503.20783/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training](https://mlsanity.com/paper/2501.17161/) (2025)
- [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025)
- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs](https://mlsanity.com/paper/2503.01307/) (2025)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2502.03373/) (2025)
- [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!](https://mlsanity.com/paper/2502.07374/) (2025)
- [Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification](https://mlsanity.com/paper/2503.06639/) (2025)
- [Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning](https://mlsanity.com/paper/2506.06632/) (2025)
- [Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning](https://mlsanity.com/paper/2504.13818/) (2025)
- [Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability](https://mlsanity.com/paper/2411.19943/) (2024)
- [Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learning](https://mlsanity.com/paper/2503.07065/) (2025)

## Built on by (most cited first, 30 of 36)

- [HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation](https://mlsanity.com/paper/2603.23871/) (2026)
- [VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL](https://mlsanity.com/paper/2511.18902/) (2025)
- [VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reduction](https://mlsanity.com/paper/2602.12579/) (2026)
- [Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning](https://mlsanity.com/paper/2605.02913/) (2026)
- [Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information](https://mlsanity.com/paper/2607.19313/) (2026)
- [GOPO: Policy Optimization using Ranked Rewards](https://mlsanity.com/paper/2602.03876/) (2026)
- [Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training](https://mlsanity.com/paper/2605.26606/) (2026)
- [Transferability for General Reasoning: An Automated Curriculum for Multi-Domain RLVR](https://mlsanity.com/paper/2606.25178/) (2026)
- [PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs](https://mlsanity.com/paper/2512.14944/) (2025)
- [Learning Agentic Policy from Action Guidance](https://mlsanity.com/paper/2605.12004/) (2026)
- [DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning](https://mlsanity.com/paper/2605.25604/) (2026)
- [sGPO: Trading Inference FLOPs for Training Efficiency in RLVR](https://mlsanity.com/paper/2606.08854/) (2026)
- [Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learning](https://mlsanity.com/paper/2608.03571/) (2026)
- [Rollout-Level Advantage-Prioritized Experience Replay for GRPO](https://mlsanity.com/paper/2606.04560/) (2026)
- [Accelerating Inference of Masked Image Generators via Reinforcement Learning](https://mlsanity.com/paper/2512.01094/) (2025)
- [Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective](https://mlsanity.com/paper/2605.12969/) (2026)
- [Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs](https://mlsanity.com/paper/2605.28388/) (2026)
- [Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR](https://mlsanity.com/paper/2606.03087/) (2026)
- [Confidence Laundering in Agent Systems: Why Uncertainty Needs a Latent Carrier](https://mlsanity.com/paper/2606.20662/) (2026)
- [CircuitLens: Reasoning Circuits as Data Selection Signals for Reinforcement Learning with Verifiable Rewards](https://mlsanity.com/paper/2609.07183/) (2026)
- [Studying the Korean Word-Chain Game with RLVR: Mitigating Reward Conflicts via Curriculum Learning](https://mlsanity.com/paper/2510.03394/) (2025)
- [Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Models](https://mlsanity.com/paper/2601.04731/) (2026)
- [Level Up: Defining and Exploiting Transitional Problems for Curriculum Learning](https://mlsanity.com/paper/2603.13761/) (2026)
- [Internalizing Curriculum Judgment for LLM Reinforcement Fine-Tuning](https://mlsanity.com/paper/2605.11235/) (2026)
- [What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA](https://mlsanity.com/paper/2605.23067/) (2026)
- [Reasoning Depth and Environment Complexity: A Controlled Study of RLVR Data Allocation across Logical Reasoning Tasks](https://mlsanity.com/paper/2605.26934/) (2026)
- [Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models](https://mlsanity.com/paper/2606.19750/) (2026)
- [LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL](https://mlsanity.com/paper/2607.04412/) (2026)
- [From Rollouts to Recipes: Self-Contained Post-Training for LLMs](https://mlsanity.com/paper/2609.01422/) (2026)
- [Anchoring What Matters: A Dual-Level Learning Framework for Visually-Grounded Multimodal Reasoning](https://mlsanity.com/paper/2609.18057/) (2026)

## Abstract (arXiv)

Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines.