mlsanityData to 14 Oct 2026

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models

arXiv 2410.18252 ↗First posted 23 Oct 2024cs.LG cs.AI cs.CLcited by 70

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The dominant paradigm for RLHF is online and on-policy RL: synchronously generating from the large language model (LLM) policy, labelling with a reward model, and learning using feedback on the LLM's own outputs. While performant, this paradigm is computationally inefficient. Inspired by classical deep RL literature, we propose separating generation and learning in RLHF. This enables asynchronous generation of new samples while simultaneously training on old samples, leading to faster training and more compute-optimal scaling. However, asynchronous training relies on an underexplored regime, online but off-policy RLHF: learning on samples from previous iterations of our model which give a worse training signal. We tackle the fundamental challenge in this regime: how much off-policyness can we tolerate for asynchronous training to speed up learning but maintain performance? Among several RLHF algorithms we test, online DPO is found to be most robust to off-policy data, and robustness increases with the scale of the policy model. We study further compute optimizations for asynchronous RLHF but find that they come at a performance cost, giving rise to a trade-off. We verify the scalability of asynchronous RLHF by training a general-purpose chatbot from LLaMA 3.1 8B on an instruction-following task ~40% faster than a synchronous run while matching final performance. Finally, we extend our results to math and reasoning to demonstrate asynchronous RL can finetune Rho 1B on GSM8k ~70% faster while matching synchronous accuracy.

Key facts. Cited by 70 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 73 references that are in the map. First posted on arXiv 23 Oct 2024.

Builds on

50 of 73 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Critique-out-Loud Reward Models21 Aug
  2. 2024Multi-turn Reinforcement Learning from Preference Human Feedback23 May
  3. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization24 Mar
  4. 2023Nash Learning from Human Feedback1 Dec
  5. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales2 Aug
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  7. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  9. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  10. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2020Learning to summarize from human feedbackcited by 2,588
  11. 2015Trust Region Policy Optimizationcited by 2,580
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
28 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  6. 2018DeepMind Control Suitecited by 956
  7. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  8. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  9. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  10. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  11. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  12. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  13. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  14. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  15. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  16. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  17. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 160
  18. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  19. 2024Rho-1: Not All Tokens Are What You Needcited by 133
  20. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  21. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  22. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 72
  23. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  24. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  25. 2023Language Model Alignment with Elastic Resetcited by 34
  26. 2022A2C is a special case of PPOcited by 16
  27. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  28. 2023Cleanba: A Reproducible and Efficient Distributed Reinforcement Learning Platformcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 30; 2026: 39.

70 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 250
  3. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 113
  4. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
  5. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  6. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  7. 2025DeepSeek-R1 Thoughtology: Let's think about LLM Reasoningcited by 92
  8. 2025Segment Policy Optimization: Effective Segment-Level Credit Assignment in RL for Large Language Modelscited by 75
  9. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  10. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  11. 2025Magistralcited by 60
  12. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
38 more
  1. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 42
  2. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  3. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  4. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  5. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generationcited by 32
  6. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 29
  7. 2025Learning to Reason at the Frontier of Learnabilitycited by 23
  8. 2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19
  9. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  10. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  11. 2025Soft Policy Optimization: Online Off-Policy RL for Sequence Modelscited by 12
  12. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 12
  13. 2026RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewardscited by 12
  14. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 11
  15. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  16. 2025MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Clustercited by 9
  17. 2025AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUscited by 9
  18. 2026Efficient RL Training for LLMs with Experience Replaycited by 9
  19. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 9
  20. 2026A Step Back: Prefix Importance Ratio Stabilizes Policy Optimizationcited by 7
  21. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  22. 2026GAC: Stabilizing Asynchronous RL Training for LLMs via Gradient Alignment Controlcited by 5
  23. 2025SeamlessFlow: A Trainer Agent Isolation RL Framework Achieving Bubble-Free Pipelines via Tag Schedulingcited by 4
  24. 2025GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learningcited by 4
  25. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  26. 2025CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Samplingcited by 3
  27. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learningcited by 3
  28. 2026ASymPO: Asymmetric-Scale Policy Optimization for Asynchronous LLM Post-Training Without Behavior Informationcited by 3
  29. 2026Trees from Marginals: Autoregressive drafting with factorized priorscited by 3
  30. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?cited by 2
  31. 2026ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learningcited by 2
  32. 2026RLHFless: Serverless Computing for Efficient RLHFcited by 2
  33. 2026AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agentscited by 2
  34. 2026Prefix Sliding for efficient test-time scalingcited by 2
  35. 2025Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learningcited by 1
  36. 2026Align and Filter: Improving Performance in Asynchronous On-Policy RLcited by 1
  37. 2026JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Trainingcited by 1
  38. 2026Experience Sharing in Mutual Reinforcement Learning for Heterogeneous Language Modelscited by 1

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation
  2. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  3. 2024Optimizing RLHF Training for Large Language Models with Stage Fusion
  4. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  5. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
  6. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  8. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL