mlsanityData to 14 Oct 2026

Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF

arXiv 2606.27580 ↗First posted 25 Jun 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) in production does not always have a synchronous reward signal. Code-execution verifiers, slow judge ensembles, and queued human review can return several gradient steps after the rollout that produced them, breaking the synchronous-reward assumption underlying standard PPO. We address this gap with Retroactive Advantage Correction (RAC): each pending slow completion is queued, aged through a non-negative kernel, and reinjected as a clipped residual into the next optimiser step's advantage. We prove that under an unbiased clipped importance ratio, the cumulative RAC correction is exactly unbiased when the effective delay kernel reinjects all of its mass, and carries a bias linear in the unreinjected fraction otherwise; at the no-delay identity kernel it reduces to V-trace. On a tabular Markov decision process (MDP) proof-of-concept, RAC reduces the closed-form policy bias by up to 47.9x at the two-slow-channel configuration, beating wait-for-slow at lower wall-clock cost. RAC integrates with PPO and GRPO through a two-line reward-manager patch.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 29 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

24 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlap30 Sep
  2. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models23 Oct
  3. 2024HybridFlow: A Flexible and Efficient RLHF Framework28 Sep
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  9. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 263
  10. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  11. 2020Reinforcement Learning with Random Delayscited by 59
  12. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
8 more
  1. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  2. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  3. 2021Off-Policy Reinforcement Learning with Delayed Rewardscited by 25
  4. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 12
  5. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 11
  6. 2022Multi-Agent Reinforcement Learning with Reward Delayscited by 9
  7. 2025Stabilizing Off-Policy Training for Long-Horizon LLM Agent via Turn-Level Importance Sampling and Clipping-Triggered Normalizationcited by 9
  8. 2026GrandCode: Achieving Grandmaster Level in Competitive Programming via Agentic Reinforcement Learningcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
  2. 2026Unifying Stable Optimization and Reference Regularization in RLHF
  3. 2026Generalisation of RLHF under Reward Shift and Clipped KL Regularisation
  4. 2025Rethinking KL Regularization in RLHF: From Value Estimation to Gradient Optimization
  5. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
  8. 2026A Gradient Perspective on RLVR Stability and Winner Advantage Policy Optimization