mlsanityData to 7 Oct 2026

Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learning

arXiv 2602.09578 ↗First posted 10 Feb 2026cs.LGcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Despite algorithm-level innovations for multi-agent reinforcement learning (MARL), the underlying networked infrastructure for large-scale MARL training remains underexplored. Existing training frameworks primarily optimize for single-agent scenarios and fail to address the unique system-level challenges of MARL, including rollout-training synchronization barriers, rollout load imbalance, and training resource underutilization. To bridge this gap, we propose FlexMARL, the first end-to-end training framework that holistically optimizes rollout, training, and their orchestration for large-scale LLM-based MARL. Specifically, FlexMARL introduces the joint orchestrator to manage data flow under the rollout-training disaggregated architecture. Building upon the experience store, a novel micro-batch driven asynchronous pipeline eliminates the synchronization barriers while providing strong consistency guarantees. Rollout engine adopts a parallel sampling scheme combined with hierarchical load balancing, which adapts to skewed inter/intra-agent request patterns. Training engine achieves on-demand hardware binding through agent-centric resource allocation. The training states of different agents are swapped via unified and location-agnostic communication. Empirical results on a large-scale production cluster demonstrate that FlexMARL achieves up to 7.3x speedup and improves hardware utilization by up to 5.6x compared to existing frameworks.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 26 of its 48 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

26 of 48 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025SkyRL-Agent: Efficient RL Training for Multi-turn LLM Agent20 Nov
  2. 2025Stronger-MAS: Multi-Agent Reinforcement Learning for Collaborative LLMs13 Oct
  3. 2025AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework5 Oct
  4. 2025HiMA-Ecom: Enabling Joint Training of Hierarchical Multi-Agent E-commerce Assistants24 Jun
  5. 2025LAMARL: LLM-Aided Multi-Agent Reinforcement Learning for Cooperative Policy Generation2 Jun
  6. 2025MARFT: Multi-Agent Reinforcement Fine-Tuning21 Apr

Other references · 20

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  5. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 903
  6. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 555
  7. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  8. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoningcited by 234
  9. 2025MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttrainingcited by 117
  10. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
  11. 2024WebPilot: A Versatile and Autonomous Multi-Agent System for Web Task Execution with Strategic Explorationcited by 100
  12. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
8 more
  1. 2025Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Librarycited by 94
  2. 2024NetLLM: Adapting Large Language Models for Networkingcited by 76
  3. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  4. 2025CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generationcited by 33
  5. 2024MAGE: A Multi-Agent Engine for Automated RTL Code Generationcited by 31
  6. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Trainingcited by 31
  7. 2025ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUscited by 10
  8. 2025Optimization without Future Compromises? Decentralized Coordination via Collective and Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026ReCrit: Transition-Aware Reinforcement Learning for Scientific Critic Reasoningcited by 0
  2. 2026Rollout Efficiency in Reinforcement Learning for Reasoning Large Language Models: A Taxonomy and Future Directionscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning
  2. 2026ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents
  3. 2025Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
  4. 2025RollArt: Disaggregated Multi-Task Agentic RL Training at Scale
  5. 2026RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
  6. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  7. 2026PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning
  8. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systems