mlsanityData to 14 Oct 2026

Align and Filter: Improving Performance in Asynchronous On-Policy RL

arXiv 2603.01365 ↗First posted 2 Mar 2026cs.LG cs.AI cs.ROcited by 1

Not on the map. Why?

Abstract (arXiv)

Distributed training and increasing the gradient update frequency are practical strategies to accelerate learning and improve performance, but both exacerbate a central challenge: textitpolicy lag, which is the mismatch between the behavior policy generating data and the learning policy being updated. Policy lag can hinder the scaling of on-policy learning algorithms to larger problems. In this paper, we identify the sources of policy lag caused by distributed learning and high update frequency. We use the findings to propose textittotal Variation-based Advantage aligned Constrained policy Optimization (methodacronym) as a practical approach to mitigate policy lag. We empirically validate our method and show that it offers better robustness to policy lag in classic RL tasks and a modern RL for LLM math reasoning task.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 48 references that are in the map. First posted on arXiv 2 Mar 2026.

Builds on

30 of 48 references.

References · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  12. 2017Constrained Policy Optimizationcited by 983
18 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  3. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  4. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  6. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  7. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 93
  8. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  9. 2019Truly Proximal Policy Optimizationcited by 69
  10. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 40
  11. 2019P3O: Policy-on Policy-off Policy Optimizationcited by 37
  12. 2024Simple Policy Optimizationcited by 32
  13. 2020Distributed Training of Deep Learning Models: A Taxonomic Perspectivecited by 25
  14. 2022Asynchronous Reinforcement Learning for Real-Time Control of Physical Robotscited by 22
  15. 2024SAPG: Split and Aggregate Policy Gradientscited by 21
  16. 2018Supervised Policy Update for Deep Reinforcement Learningcited by 11
  17. 2024Acceleration for Deep Reinforcement Learning using Parallel and Distributed Computing: A Surveycited by 8
  18. 2023Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platformcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Control
  2. 2026ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information
  3. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learning
  4. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning
  5. 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
  6. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
  7. 2025Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
  8. 2025Adaptive Scaling of Policy Constraints for Offline Reinforcement Learning