# Laminar: A Scalable Asynchronous RL Post-Training Framework

arXiv: https://arxiv.org/abs/2510.12633 · HTML: https://mlsanity.com/paper/2510.12633/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 104 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 30 of 57)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates](https://mlsanity.com/paper/1610.00633/) (2016)
- [Tool Learning with Foundation Models](https://mlsanity.com/paper/2304.08354/) (2023)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [Llumnix: Dynamic Scheduling for Large Language Model Serving](https://mlsanity.com/paper/2406.03243/) (2024)
- [Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning](https://mlsanity.com/paper/2504.13914/) (2025)
- [LLM Inference Unveiled: Survey and Roofline Model Insights](https://mlsanity.com/paper/2402.16363/) (2024)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs](https://mlsanity.com/paper/2402.15627/) (2024)
- [DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales](https://mlsanity.com/paper/2308.01320/) (2023)
- [NanoFlow: Towards Optimal Large Language Model Serving Throughput](https://mlsanity.com/paper/2408.12757/) (2024)

## Built on by (most cited first, 30 of 37)

- [Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning](https://mlsanity.com/paper/2511.14617/) (2025)
- [RollArt: Disaggregated Multi-Task Agentic RL Training at Scale](https://mlsanity.com/paper/2512.22560/) (2025)
- [CONCUR: High-Throughput Agentic Batch Inference of LLM via Congestion-Based Concurrency Control](https://mlsanity.com/paper/2601.22705/) (2026)
- [DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training](https://mlsanity.com/paper/2604.26256/) (2026)
- [A Step Back: Prefix Importance Ratio Stabilizes Policy Optimization](https://mlsanity.com/paper/2601.22718/) (2026)
- [RL-VLA3: A Flexible and Asynchronous Reinforcement Learning Framework for VLA Training](https://mlsanity.com/paper/2602.05765/) (2026)
- [Diagnosing Training Inference Mismatch in LLM Reinforcement Learning via a Zero-Mismatch Reference](https://mlsanity.com/paper/2605.14220/) (2026)
- [Missing Old Logits in Asynchronous Agentic RL: Semantic Mismatch and Repair Methods for Off-Policy Correction](https://mlsanity.com/paper/2605.12070/) (2026)
- [TensorHub: Scalable and Elastic Weight Transfer for LLM RL Training](https://mlsanity.com/paper/2604.09107/) (2026)
- [Libra: Efficient Resource Management for Agentic RL Post-Training](https://mlsanity.com/paper/2606.03077/) (2026)
- [TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling](https://mlsanity.com/paper/2608.10402/) (2026)
- [FlashEvolve: Accelerating Agent Self-Evolution with Asynchronous Stage Orchestration](https://mlsanity.com/paper/2605.08520/) (2026)
- [AsyncOPD: How Stale Can On-Policy Distillation Be?](https://mlsanity.com/paper/2606.24143/) (2026)
- [WAR: Workload-Aware Rollouts for Synchronous Agentic Reinforcement Learning](https://mlsanity.com/paper/2607.17299/) (2026)
- [Role-Based Fault Tolerance System for LLM RL Post-Training](https://mlsanity.com/paper/2512.22492/) (2025)
- [ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning](https://mlsanity.com/paper/2602.02192/) (2026)
- [DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning](https://mlsanity.com/paper/2605.30859/) (2026)
- [Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training](https://mlsanity.com/paper/2606.11867/) (2026)
- [SimpleSearch-VL: A Simple Recipe for Multimodal Agentic Deep Search](https://mlsanity.com/paper/2606.31504/) (2026)
- [DynaResize: Runtime GPU Reallocation for Disaggregated LLM Post-Training](https://mlsanity.com/paper/2607.22614/) (2026)
- [Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learning](https://mlsanity.com/paper/2511.18871/) (2025)
- [AREAL-DTA: Dynamic Tree Attention for Efficient Reinforcement Learning of Large Language Models](https://mlsanity.com/paper/2602.00482/) (2026)
- [AcceRL: A Distributed Asynchronous Reinforcement Learning and World Model Framework for Vision-Language-Action Models](https://mlsanity.com/paper/2603.18464/) (2026)
- [JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training](https://mlsanity.com/paper/2604.23838/) (2026)
- [MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service](https://mlsanity.com/paper/2605.08527/) (2026)
- [MinT: Managed Infrastructure for Training and Serving Millions of LLMs](https://mlsanity.com/paper/2605.13779/) (2026)
- [Moebius: Serving Mixture-of-Expert Models with Seamless Runtime Parallelism Switch](https://mlsanity.com/paper/2606.26607/) (2026)
- [Nereus: Adaptive Parallelism for LLM Post-Training](https://mlsanity.com/paper/2609.34645/) (2026)
- [Demystifying Design Choices of Reinforcement Fine-tuning: A Batched Contextual Bandit Learning Perspective](https://mlsanity.com/paper/2601.22532/) (2026)
- [Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Training](https://mlsanity.com/paper/2606.19004/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) post-training for Large Language Models (LLMs) is now scaling to large clusters and running for extended durations to enhance model reasoning performance. However, the scalability of existing RL frameworks is limited, as extreme long-tail skewness in RL trajectory generation causes severe GPU underutilization. Current asynchronous RL systems attempt to mitigate this, but they rely on global weight synchronization between the actor and all rollouts, which creates a rigid model update schedule. This global synchronization is ill-suited for the highly skewed and evolving distribution of trajectory generation latency in RL training, crippling training efficiency. Our key insight is that efficient scaling requires breaking this lockstep through trajectory-level asynchrony, which generates and consumes each trajectory independently. We propose Laminar, a scalable and robust RL post-training system built on a fully decoupled architecture. First, we replace global updates with a tier of relay workers acting as a distributed parameter service. This enables asynchronous and fine-grained weight synchronization, allowing rollouts to pull the latest weight anytime without stalling the actor's training loop. Second, a dynamic repack mechanism consolidates long-tail trajectories onto a few dedicated rollouts, maximizing generation throughput. The fully decoupled design also isolates failures, ensuring robustness for long-running jobs. Our evaluation on a 1024-GPU cluster shows that Laminar achieves up to 5.48× training throughput speedup over state-of-the-art systems, while reducing model convergence time.