mlsanityData to 14 Oct 2026

AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning

arXiv 2505.24298 ↗First posted 30 May 2025cs.LG cs.AIcited by 250

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has become a dominant paradigm for training large language models (LLMs), particularly for reasoning tasks. Effective RL for LLMs requires massive parallelization and poses an urgent need for efficient training systems. Most existing large-scale RL systems for LLMs are synchronous, alternating generation and training in a batch setting where rollouts in each training batch are generated by the same model. This approach stabilizes RL training but suffers from severe system-level inefficiency: generation must wait until the longest output in the batch is completed before model updates, resulting in GPU underutilization. We present AReaL, a fully asynchronous RL system that completely decouples generation from training. Rollout workers in AReaL continuously generate new outputs without waiting, while training workers update the model whenever a batch of data is collected. AReaL also incorporates a collection of system-level optimizations, leading to substantially higher GPU utilization. To stabilize RL training, AReaL balances the workload of rollout and training workers to control data staleness, and adopts a staleness-enhanced PPO variant to better handle outdated training samples. Extensive experiments on math and code reasoning benchmarks show that AReaL achieves up to 2.77× training speedup compared to synchronous systems with the same number of GPUs and matched or improved final performance. The code of AReaL is available at https://github.com/inclusionAI/AReaL/.

Key facts. Cited by 250 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 63 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

41 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025INTELLECT-2: A Reasoning Model Trained Through Globally Decentralized Reinforcement Learning12 May
  2. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  3. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training24 Mar
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2024Qwen2.5 Technical Reportcited by 4,942
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
23 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  5. 2025s1: Simple test-time scalingcited by 1,341
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  10. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  13. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  14. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  15. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  16. 2024DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Datacited by 234
  17. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 201
  18. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  19. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  20. 2025StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generationcited by 61
  21. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 41
  22. 2021Batch size-invariance for policy optimizationcited by 35
  23. 2023SRL: Scaling Distributed Reinforcement Learning to Over Ten Thousand Corescited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 70; 2026: 180.

250 papers.

Sort:

By area:

  1. 2025LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Trainingcited by 244
  2. 2026MiMo-V2-Flash Technical Reportcited by 191
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  4. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 150
  5. 2025Reinforcement Learning with Rubric Anchorscited by 113
  6. 2025Agent Lightning: Train ANY AI Agents with Reinforcement Learningcited by 109
  7. 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
  8. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 90
  9. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Frameworkcited by 62
  10. 2025VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agentscited by 62
  11. 2025RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformationcited by 57
  12. 2025Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Modelcited by 55
38 more
  1. 2026LLaDA2.1: Speeding Up Text Diffusion via Token Editingcited by 54
  2. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?cited by 53
  3. 2025Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Managementcited by 51
  4. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 50
  5. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agentcited by 48
  6. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  7. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 45
  8. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 44
  9. 2025Continuum: Efficient and Robust Multi-Turn LLM Agent Scheduling with KV Cache Time-to-Livecited by 44
  10. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 41
  11. 2025On Entropy Control in LLM-RL Algorithmscited by 40
  12. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimizationcited by 39
  13. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
  14. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 37
  15. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 37
  16. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 35
  17. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 34
  18. 2025APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generationcited by 30
  19. 2025Agent-R1: A Unified and Modular Framework for Agentic Reinforcement Learningcited by 29
  20. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Modelscited by 28
  21. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 26
  22. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Rewardcited by 25
  23. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 25
  24. 2025Introducing LongCat-Flash-Thinking: A Technical Reportcited by 22
  25. 2026From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agentscited by 21
  26. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  27. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 18
  28. 2026Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillationcited by 18
  29. 2025AWorld: Orchestrating the Training Recipe for Agentic AIcited by 17
  30. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  31. 2026ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference Systemcited by 17
  32. 2025CodeDance: A Dynamic Tool-integrated MLLM for Executable Visual Reasoningcited by 16
  33. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  34. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 16
  35. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchronycited by 15
  36. 2025Thinking in 360deg: Humanoid Visual Search in the Wildcited by 15
  37. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 14
  38. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Draftercited by 14

50 most cited of 250 All 250 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Training
  2. 2026SortedRL: Accelerating RL Training for LLMs through Online Length-Aware Scheduling
  3. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  4. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
  5. 2026A Survey of Reinforcement Learning for Large Language Models under Data Scarcity: Challenges and Solutions
  6. 2025EARL: Efficient Agentic Reinforcement Learning Systems for Large Language Models
  7. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
  8. 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement