mlsanityData to 7 Oct 2026

DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training

arXiv 2604.26256 ↗First posted 29 Apr 2026cs.LG cs.DCcited by 13

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) has become a critical paradigm for LLM post-training, yet the rollout phase -- accounting for 50--80% of total step time -- is bottlenecked by skewed generation: long-tailed trajectories indispensable for model performance block the entire training pipeline. Asynchronous training offers a natural remedy by overlapping generation with training, but introduces a fundamental tension between efficiency and algorithmic correctness. We identify three constraints in asynchronous training to preserve convergence: intra-trajectory policy consistency, data integrity, and bounded staleness. Existing approaches fail to intrinsically address the long-tailed trajectory problem, which is further exacerbated by the imbalance characteristic of Mix-of-Experts models, or deviate from the standard RL training formulation, thereby hindering model convergence. Therefore, we propose DORA (Dynamic ORchestration for Asynchronous Rollout), which addresses this challenge through algorithm-system co-design. DORA introduces multi-version streaming rollout, a novel asynchronous paradigm that maintains multiple policy versions concurrently -- simultaneously achieving full bubble elimination without compromising algorithmic constraints. Experimental results demonstrate that our DORA system achieves substantial improvements in throughput -- up to 2--3 times higher than state-of-the-art systems on open-source benchmarks -- without compromising convergence. Furthermore, in large-scale industrial applications with tens of thousands of accelerators, DORA accelerates RL training by 2--4 times compared to synchronous training across various scenarios. The resultant open-source models, LongCat-Flash-Thinking, exhibit competitive performance on complex reasoning benchmarks, matching the capability of most advanced LLMs.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 44 of its 54 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

44 of 54 references.

References · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2024Mixtral of Expertscited by 1,785
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
32 more
  1. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  2. 2017Constrained Policy Optimizationcited by 928
  3. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  4. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  5. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
  6. 2022Efficiently Scaling Transformer Inferencecited by 529
  7. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  8. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  9. 2025Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learningcited by 159
  10. 2015Asynchronous Parallel Stochastic Gradient for Nonconvex Optimizationcited by 115
  11. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
  12. 2016Asynchronous Stochastic Gradient Descent with Delay Compensationcited by 98
  13. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 84
  14. 2025DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoningcited by 84
  15. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  16. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 56
  17. 2025LongCat-Flash Technical Reportcited by 54
  18. 2025RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformationcited by 51
  19. 2025VitaBench: Benchmarking LLM Agents with Versatile Interactive Tasks in Real-world Applicationscited by 45
  20. 2021Generalized Proximal Policy Optimization with Sample Reusecited by 39
  21. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  22. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  23. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 35
  24. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  25. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 32
  26. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 31
  27. 2025On-Policy RL with Optimal Reward Baselinecited by 30
  28. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 21
  29. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13
  30. 2025UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilitiescited by 11
  31. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Schedulingcited by 4
  32. 2025SeamlessFlow: A Trainer Agent Isolation RL Framework Achieving Bubble-Free Pipelines via Tag Schedulingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026FrogNano: Training a 4B Coding Agent via Online Task Synthesiscited by 3
  2. 2026Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switchcited by 1
  3. 2026Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learningcited by 1
  4. 2026Scheduling Mixed RL Rollouts Beyond Prefix Localitycited by 1
  5. 2026Molt: A Scalable PyTorch-Native Training Framework for Agentic Reinforcement Learningcited by 0
  6. 2026Deconstructing Off-Policy Ratios: Entropy-Normalized Trust Regions for Asynchronous Reinforcement Learningcited by 0
  7. 2026BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RLcited by 0
  8. 2026PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Servingcited by 0
  9. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0
  10. 2026QwenGyre: An Elastic Reinforcement Learning Framework for Training xLong-Horizon Agentscited by 0
  11. 2026DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agentscited by 0
  12. 2026Where Does Staleness Accumulate? Pool Aware Effective Staleness Control for Asynchronous RL in LLM Post-Trainingcited by 0
1 more
  1. 2026Asynchronous LLM Post-Training: Group-Mass Capping and Convergence Analysiscited by 0