mlsanityData to 14 Oct 2026

LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training

arXiv 2505.24034 ↗First posted 29 May 2025cs.LG cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) has become the most effective post-training approach for improving the capabilities of Large Language Models (LLMs). In practice, because of the high demands on latency and memory, it is particularly challenging to develop an efficient RL framework that reliably manages policy models with hundreds to thousands of billions of parameters. In this paper, we present LlamaRL, a fully distributed, asynchronous RL framework optimized for efficient training of large-scale LLMs with various model sizes (8B, 70B, and 405B parameters) on GPU clusters ranging from a handful to thousands of devices. LlamaRL introduces a streamlined, single-controller architecture built entirely on native PyTorch, enabling modularity, ease of use, and seamless scalability to thousands of GPUs. We also provide a theoretical analysis of LlamaRL's efficiency, including a formal proof that its asynchronous design leads to strict RL speed-up. Empirically during the Llama 3 post-training, by leveraging best practices such as colocated model offloading, asynchronous off-policy training, and distributed direct memory access for weight synchronization, LlamaRL achieves significant efficiency gains -- up to 10.7x speed-up compared to DeepSpeed-Chat-like systems on a 405B-parameter policy model. Furthermore, the efficiency advantage continues to grow with increasing model scale, demonstrating the framework's suitability for future large-scale RL training.

Key facts. Cited by 42 papers in the map (data through 14 Oct 2026). Builds on 48 of its 68 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

48 of 68 references.

References · 48

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  9. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  10. 2020Scaling Laws for Neural Language Modelscited by 6,586
  11. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  12. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
36 more
  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2023Let's Verify Step by Stepcited by 3,789
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2015Trust Region Policy Optimizationcited by 2,580
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  10. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  13. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  14. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  15. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  16. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  17. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  18. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  19. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  20. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  21. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  22. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  23. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  24. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  25. 2024When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Methodcited by 204
  26. 2016Sample Efficient Actor-Critic with Experience Replaycited by 188
  27. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 187
  28. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  29. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  30. 2025Competitive Programming with Large Reasoning Modelscited by 119
  31. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
  32. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  33. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  34. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  35. 2019ELF OpenGo: An Analysis and Open Reimplementation of AlphaZerocited by 39
  36. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 28.

42 papers.

Sort:

By area:

  1. 2025Magistralcited by 60
  2. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  3. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  4. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 30
  5. 2025Single-stream Policy Optimizationcited by 28
  6. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 22
  7. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 15
  8. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 14
  9. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 12
  10. 2026Stable Asynchrony: Variance-Controlled Off-Policy RL for LLMscited by 11
  11. 2026Efficient RL Training for LLMs with Experience Replaycited by 9
  12. 2026Accelerating RL Post-Training Rollouts via System-Integrated Speculative Decodingcited by 9
30 more
  1. 2025LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMscited by 5
  2. 2026Jackpot: Optimal Budgeted Rejection Sampling for Extreme Actor-Policy Mismatch Reinforcement Learningcited by 5
  3. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Trainingcited by 5
  4. 2026Sparrow: Sparse Rollout for Stable and Efficient Long-context RL of Large Language Modelscited by 5
  5. 2025GEPO: Group Expectation Policy Optimization for Stable Heterogeneous Reinforcement Learningcited by 4
  6. 2026TensorHub: Scalable and Elastic Weight Transfer for LLM RL Trainingcited by 4
  7. 2026Libra: Efficient Resource Management for Agentic RL Post-Trainingcited by 4
  8. 2025Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharingcited by 3
  9. 2025Hail to the Thief: Exploring Attacks and Defenses in Decentralised GRPOcited by 2
  10. 2025Role-Based Fault Tolerance System for LLM RL Post-Trainingcited by 2
  11. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPOcited by 2
  12. 2026Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-trainingcited by 2
  13. 2026Staleness-Learning Rate Scaling Laws for Asynchronous RLHFcited by 2
  14. 2025Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learningcited by 1
  15. 2026CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Productioncited by 1
  16. 2026AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Modelscited by 1
  17. 2026MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Servicecited by 1
  18. 2026F-TIS: Harnessing Diverse Models in Collaborative GRPOcited by 1
  19. 2026Nereus: Adaptive Parallelism for LLM Post-Trainingcited by 1
  20. 2026Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspectivecited by 0
  21. 2026When to Stop Reusing: Dynamic Gradient Gating for Sample-Efficient RLVRcited by 0
  22. 2026Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generationcited by 0
  23. 2026Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learningcited by 0
  24. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0
  25. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  26. 2026Not Every Token Is Worth Distilling: Selective Supervision for Direct-OPDcited by 0
  27. 2026ThunderSyncRL: Lossless Acceleration of Agentic Reinforcement Learningcited by 0
  28. 2026COPC: Coupled Off-Policy Correction for Asynchronous LLM Reinforcement Learningcited by 0
  29. 2026Target-ESS Tempered Policy Optimization (TETPO): Smooth Variance Control via Bridge Importance Samplingcited by 0
  30. 2026Efficient Resource-Shiftable RL Post-training with Generative Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
  2. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
  3. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
  4. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
  5. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
  6. 2026AuroraRL: Fast, Fault-Tolerant, and Cost-Efficient Reinforcement Learning over Decentralized Network
  7. 2025The Art of Scaling Reinforcement Learning Compute for LLMs
  8. 2026A Production-Grade, High-Efficiency Agentic RL Framework for Frontier LLMs with Validated Recipes