CVPO: Enhancing LLM Reinforcement Learning Reasoning via Value-Variance Adaptation and Dynamic Curriculum Learning
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement learning (RL) has emerged as an effective method for enhancing the reasoning capabilities of large language models (LLMs). However, existing methods suffer from insufficient precision in feedback on generated answer trajectories and exhibit the phenomenon of problem difficulty drift. To address these challenges, we propose CVPO - Curriculum-guided Value-Variance Policy Optimization. At the response trajectory level, we find that token-level value-variance correlates with exploration intensity. Our theoretical analysis shows this variance bounds policy update magnitude. We then use the estimated trajectory value-variance to quantify the intrinsic randomness in generation. Based on this, we design a variance-aware advantage adjustment mechanism for different reward types. At the question level, we introduce a dynamic curriculum weighting method that adapts to question difficulty. This helps the model focus on tasks matched to its current ability during each training stage. Experimental results show our method outperforms strong value-based baselines like VAPO. It achieves better performance and stronger exploration, enabling more accurate and robust reasoning in language models across various math tasks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 20 references that are in the map. First posted on arXiv 4 Aug 2026.
Builds on
18 of 20 references.
Closest ancestors · 13
Same direction, newest first
- 2025Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?7 Jul
- 2025Spurious Rewards: Rethinking Training Signals in RLVR12 Jun
- 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning7 Jun
- 2025Maximizing Confidence Alone Improves Reasoning28 May
- 2025DGRO: Enhancing LLM Reasoning via Exploration-Exploitation Control and Reward Variance Management19 May
- 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
- 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
- 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
- 2025What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret3 Mar
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
- 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
1 more
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 5
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2024DeepSeek-V3 Technical Reportcited by 724
- 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
- 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
- 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
- 2025CLPO: Curriculum Learning meets Policy Optimization for LLM Reasoning
- 2025Self-Evolving Curriculum for LLM Reasoning
- 2026VI-CuRL: Stabilizing Verifier-Independent RL Reasoning via Confidence-Guided Variance Reduction
- 2025Towards Analyzing and Understanding the Limitations of VAPO: A Theoretical Perspective
- 2025VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training
- 2025Learning to Reason at the Frontier of Learnability
- 2026Selector-Guided Autonomous Curriculum for One-Shot Reinforcement Learning from Verifiable Rewards