mlsanityData to 7 Oct 2026

Staleness-Learning Rate Scaling Laws for Asynchronous RLHF

arXiv 2607.01083 ↗First posted 1 Jul 2026cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Asynchronous reinforcement learning improves the throughput of large language model post-training by decoupling rollout generation from policy optimization, but introduces a mismatch between the behavior and learner policies. How the resulting policy staleness couples with the learning rate to govern training stability and collapse time remains poorly understood. We investigate this coupling in vanilla GRPO through controlled sweeps of the synchronization interval S and constant learning rate η on Llama-3.2-1B/3B, complemented by experiments on Qwen3-8B. We identify two empirical scaling laws: (i) Stability-boundary scaling: the largest stable learning rate scales approximately as S-1, yielding a stability boundary characterized by an approximately constant product Sη. (ii) Collapse-time scaling: among collapsing runs, estimated collapse times scale approximately as η-1, corresponding to a model- and setup-dependent cumulative learning-rate budget that aligns across synchronization intervals in the Llama sweeps. We interpret these laws through a local analysis of the behavior-dependent GRPO surrogate and a complementary mean-field model. Under local regularity conditions, the analysis yields an O(Sη) upper bound on the staleness-induced update bias that resets at synchronization. The mean-field model shows how sufficiently strong positive feedback can sustain directional drift when update directions persist across synchronization cycles. When drift speed saturates under optimizer normalization, this mechanism predicts exit from a local surrogate-validity region after an approximately fixed cumulative learning rate. Together, these findings motivate a practical calibration rule: estimate the stability threshold and collapse budget from a coarse sweep, then jointly select S and η for the intended training horizon.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 54 references that are in the map. First posted on arXiv 1 Jul 2026.

Builds on

45 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024HybridFlow: A Flexible and Efficient RLHF Framework28 Sep

Other references · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
32 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
  3. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  4. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  6. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  7. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  8. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  9. 2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
  10. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  11. 2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
  12. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 100
  13. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  14. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  15. 2018Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGDcited by 68
  16. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  17. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  18. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 49
  19. 2015Staleness-aware Async-SGD for Distributed Deep Learningcited by 43
  20. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  21. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  22. 2021Batch size-invariance for policy optimizationcited by 33
  23. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Trainingcited by 33
  24. 2016Asynchrony begets Momentum, with an Application to Deep Learningcited by 30
  25. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 29
  26. 2025Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friendscited by 13
  27. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 11
  28. 2023Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platformcited by 7
  29. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learningcited by 5
  30. 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Controlcited by 4
  31. 2026ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learningcited by 2
  32. 2025Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learningcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMs
  2. 2026Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Training
  3. 2026Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysis
  4. 2026Stale but Stable: Staleness-Adaptive Trust Regions for Stabilizing Asynchronous Reinforcement Learning
  5. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  6. 2026ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Information
  7. 2026RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
  8. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPO