mlsanityData to 7 Oct 2026

Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony

arXiv 2510.11345 ↗First posted 13 Oct 2025cs.LG cs.AIcited by 13

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing Large Language Models (LLMs) with diverse capabilities. However, many systems designed to accelerate RL post-training still suffer from low resource utilization and limited scalability. We present ROLL Flash, a system that extends ROLL with native support for asynchronous RL post-training. ROLL Flash is built upon two core design principles: fine-grained parallelism and rollout-train decoupling. Guided by these principles, ROLL Flash provides flexible programming interfaces that enable a fully asynchronous training architecture and support efficient rollout mechanisms, including queue scheduling and environment-level asynchronous execution. Through comprehensive theoretical analysis and extensive experiments, we demonstrate that ROLL Flash significantly improves resource utilization and scalability over synchronous RL post-training. ROLL Flash achieves up to 2.24x speedup on RLVR tasks and 2.72x on agentic tasks, using the same GPU budget as synchronous baselines. Furthermore, we implement several popular off-policy algorithms and verify that asynchronous training can achieve performance on par with synchronous training.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 29 of its 36 references that are in the map. First posted on arXiv 13 Oct 2025.

Builds on

29 of 36 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May

Other references · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
16 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 300
  3. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  5. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  6. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  7. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  8. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 112
  9. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  10. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  11. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  12. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  13. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 35
  14. 2021Batch size-invariance for policy optimizationcited by 33
  15. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 31
  16. 2022The Sufficiency of Off-Policyness and Soft Clipping: PPO is still Insufficient according to an Off-Policy Measurecited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 9.

13 papers.

Sort:

By area:

  1. 2026LongCat-Flash-Thinking-2601 Technical Reportcited by 32
  2. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 26
  3. 2025Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimizationcited by 23
  4. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  5. 2025AgentMath: Empowering Mathematical Reasoning for Large Language Models via Tool-Augmented Agentcited by 17
  6. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scalecited by 11
  7. 2026Freshness-Aware Prioritized Experience Replay for LLM/VLM Reinforcement Learningcited by 3
  8. 2026How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMscited by 3
  9. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 2
  10. 2026DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learningcited by 2
  11. 2026Apodex 1.1: Scaling Agentic Intelligence for Complex Workcited by 2
  12. 2026AgentJet: A Distributed Swarm Training Framework for Agentic Reinforcement Learningcited by 1
1 more
  1. 2026Cool the Sampler, Not the Learner: Sampling Temperature Moves the Staleness Cliff of Importance-Corrected GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Libra: Efficient Resource Management for Agentic RL Post-Training
  2. 2025Periodic Asynchrony: An On-Policy Approach for Accelerating LLM Reinforcement Learning
  3. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  4. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training
  5. 2026Spend Your Rollouts Where It Counts: Rollout Allocation for Group-Based RL Post-Training
  6. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
  7. 2025SPEC-RL: Accelerating On-Policy Reinforcement Learning with Speculative Rollouts
  8. 2026ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning