mlsanityData to 14 Oct 2026

ABC: Advantage-Based Control Variates for Reinforcement Learning with Verifiable Rewards

arXiv 2609.36058 ↗First posted 28 Sep 2026cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Recent progress in reinforcement learning with verifiable rewards (RLVR) has highlighted the effectiveness of simple critic-free policy-gradient methods such as Group Relative Policy Optimization (GRPO). In contrast, actor-critic methods rely on learned value functions whose approximation error can introduce bias through commonly used advantage estimators such as temporal-difference error. Motivated by this observation, we revisit trajectory-level control variates through an advantage-value formulation, which we call Advantage-Based Control Variates (ABC). This formulation reveals that the covariance structure is closely related to the return decomposition used in Direct Advantage Estimation (DAE). Finally, we combine ABC with DAE into a single actor-critic algorithm and evaluate it in an offline-to-online RLVR setting, where the critic is first trained on previously collected trajectories and adapted during online learning. On mathematical reasoning tasks, ABC achieves performance competitive with GRPO using substantially fewer online optimization steps.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 28 of its 36 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

28 of 36 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation23 Sep
  2. 2025RePO: Replay-Enhanced Policy Optimization11 Jun
  3. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoning15 Apr
  4. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Training22 Nov
  9. 2023Let's Verify Step by Step31 May
  10. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 18

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
  9. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  10. 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 187
  11. 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 81
  12. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
6 more
  1. 2020Counterfactual Credit Assignment in Model-Free Reinforcement Learningcited by 64
  2. 2019Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methodscited by 15
  3. 2021Direct Advantage Estimationcited by 11
  4. 2024Skill or Luck? Return Decomposition via Advantage Functionscited by 8
  5. 2026On the Variance of Temporal Difference Learning and its Reduction Using Control Variatescited by 1
  6. 2026Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On Advantage Estimates for Max@K Policy Gradients
  2. 2025Taming OOD Actions for Offline Reinforcement Learning: An Advantage-Based Approach
  3. 2026DVAO: Dynamic Variance-adaptive Advantage Optimization for Multi-reward Reinforcement Learning
  4. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  5. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification
  6. 2026EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training
  7. 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
  8. 2026Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation