mlsanityData to 7 Oct 2026

MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning

arXiv 2106.07551 ↗First posted 5 Jun 2021cs.MA cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

Population-based multi-agent reinforcement learning (PB-MARL) refers to the series of methods nested with reinforcement learning (RL) algorithms, which produces a self-generated sequence of tasks arising from the coupled population dynamics. By leveraging auto-curricula to induce a population of distinct emergent strategies, PB-MARL has achieved impressive success in tackling multi-agent tasks. Despite remarkable prior arts of distributed RL frameworks, PB-MARL poses new challenges for parallelizing the training frameworks due to the additional complexity of multiple nested workloads between sampling, training and evaluation involved with heterogeneous policy interactions. To solve these problems, we present MALib, a scalable and efficient computing framework for PB-MARL. Our framework is comprised of three key components: (1) a centralized task dispatching model, which supports the self-generated tasks and scalable training with heterogeneous policy combinations; (2) a programming architecture named Actor-Evaluator-Learner, which achieves high parallelism for both training and sampling, and meets the evaluation requirement of auto-curriculum learning; (3) a higher-level abstraction of MARL training paradigms, which enables efficient code reuse and flexible deployments on different distributed computing paradigms. Experiments on a series of complex tasks such as multi-agent Atari Games show that MALib achieves throughput higher than 40K FPS on a single machine with 32 CPU cores; 5x speedup than RLlib and at least 3x speedup than OpenSpiel in multi-agent training tasks. MALib is publicly available at https://github.com/sjtu-marl/malib.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 39 of its 68 references that are in the map. First posted on arXiv 5 Jun 2021.

Builds on

39 of 68 references.

References · 39

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  7. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2018QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 697
  11. 2019The StarCraft Multi-Agent Challengecited by 589
  12. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
27 more
  1. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  2. 2017Population Based Training of Neural Networkscited by 351
  3. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 345
  4. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  5. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  6. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  7. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  8. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  9. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 257
  10. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 209
  11. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
  12. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  13. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  14. 2020Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learningcited by 84
  15. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 80
  16. 2019A Generalized Training Approach for Multiagent Learningcited by 53
  17. 2019α-Rank: Multi-Agent Evaluation by Evolutioncited by 50
  18. 2020Pipeline PSRO: A Scalable Approach for Finding Approximate Nash Equilibria in Large Gamescited by 47
  19. 2016Reinforcement Learning through Asynchronous Advantage Actor-Critic on a GPUcited by 46
  20. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40
  21. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  22. 2019Multi-Agent Reinforcement Learning for Order-dispatching via Order-Vehicle Distribution Matchingcited by 26
  23. 2019Decentralized Multi-Agent Reinforcement Learning with Networked Agents: Recent Advancescited by 23
  24. 2017ELF: An Extensive, Lightweight and Flexible Research Platform for Real-time Strategy Gamescited by 16
  25. 2020TLeague: A Framework for Competitive Self-Play based Distributed Multi-Agent Reinforcement Learningcited by 11
  26. 2019αα-Rank: Practically Scaling α-Rank through Stochastic Optimisationcited by 10
  27. 2020Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methodscited by 4

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 6; 2023: 6; 2024: 7; 2025: 1.

23 papers.

Sort:

By area:

  1. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 180
  2. 2023Heterogeneous-Agent Reinforcement Learningcited by 67
  3. 2023JaxMARL: Multi-Agent RL Environments and Algorithms in JAXcited by 59
  4. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  5. 2024Learning Multi-Agent Communication from Graph Modeling Perspectivecited by 47
  6. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  7. 2024Multi-Agent Reinforcement Learning for Autonomous Driving: A Surveycited by 30
  8. 2023Order Matters: Agent-by-agent Policy Optimizationcited by 24
  9. 2022MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Librarycited by 21
  10. 2022An Introduction to Multi-Agent Reinforcement Learning and Review of its Application to Autonomous Mobilitycited by 16
  11. 2024FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learningcited by 13
  12. 2021Measuring the Non-Transitivity in Chesscited by 10
11 more
  1. 2024Policy Space Response Oracles: A Surveycited by 10
  2. 2023Pearl: A Production-ready Reinforcement Learning Agentcited by 9
  3. 2022Understanding Value Decomposition Algorithms in Deep Cooperative Multi-Agent Reinforcement Learningcited by 6
  4. 2023An Empirical Study on Google Research Football Multi-agent Scenarioscited by 5
  5. 2024Safe Multi-Agent Reinforcement Learning with Convergence to Generalized Nash Equilibriumcited by 5
  6. 2022MSRL: Distributed Reinforcement Learning with Dataflow Fragmentscited by 4
  7. 2022Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolboxcited by 4
  8. 2022Lamarckian Platform: Pushing the Boundaries of Evolutionary Reinforcement Learning towards Asynchronous Commercial Gamescited by 3
  9. 2023marl-jax: Multi-Agent Reinforcement Leaning Frameworkcited by 1
  10. 2024PyTAG: Tabletop Games for Multi-Agent Reinforcement Learningcited by 1
  11. 2024Communication Learning in Multi-Agent Systems from Graph Modeling Perspectivecited by 0