# MARLlib: A Scalable and Efficient Multi-agent Reinforcement Learning Library

arXiv: https://arxiv.org/abs/2210.13708 · HTML: https://mlsanity.com/paper/2210.13708/

## Key facts

- Cited by 21 papers in the map (data through 7 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 11 Oct 2022.

## Builds on (most cited first, 24 of 24)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [Emergent Tool Use From Multi-Agent Autocurricula](https://mlsanity.com/paper/1909.07528/) (2019)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [PettingZoo: Gym for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2009.14471/) (2020)
- [Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks](https://mlsanity.com/paper/2006.07869/) (2020)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2109.11251/) (2021)
- [FACMAC: Factored Multi-Agent Centralised Policy Gradients](https://mlsanity.com/paper/2003.06709/) (2020)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2006.07169/) (2020)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [Game-Theoretic Multiagent Reinforcement Learning](https://mlsanity.com/paper/2011.00583/) (2020)
- [MAgent: A Many-Agent Reinforcement Learning Platform for Artificial Collective Intelligence](https://mlsanity.com/paper/1712.00600/) (2017)
- [Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.03479/) (2021)
- [Value-Decomposition Multi-Agent Actor-Critics](https://mlsanity.com/paper/2007.12306/) (2020)
- [Pommerman: A Multi-Agent Playground](https://mlsanity.com/paper/1809.07124/) (2018)
- [MALib: A Parallel Framework for Population-based Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2106.07551/) (2021)
- [The Neural MMO Platform for Massively Multiagent Research](https://mlsanity.com/paper/2110.07594/) (2021)

## Built on by (most cited first, 21 of 21)

- [JaxMARL: Multi-Agent RL Environments and Algorithms in JAX](https://mlsanity.com/paper/2311.10090/) (2023)
- [BenchMARL: Benchmarking Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2312.01472/) (2023)
- [GEM: A Gym for Agentic LLMs](https://mlsanity.com/paper/2510.01051/) (2025)
- [FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2406.02081/) (2024)
- [Pearl: A Production-ready Reinforcement Learning Agent](https://mlsanity.com/paper/2312.03814/) (2023)
- [MAexp: A Generic Platform for RL-based Multi-Agent Exploration](https://mlsanity.com/paper/2404.12824/) (2024)
- [Major-Minor Mean Field Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.10665/) (2023)
- [A Reference Architecture of Reinforcement Learning Frameworks](https://mlsanity.com/paper/2603.06413/) (2026)
- [Multi-Agent Reinforcement Learning in Cybersecurity: From Fundamentals to Applications](https://mlsanity.com/paper/2505.19837/) (2025)
- [Cooperative Game-Theoretic Credit Assignment for Multi-Agent Policy Gradients via the Core](https://mlsanity.com/paper/2506.04265/) (2025)
- [GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspective](https://mlsanity.com/paper/2507.09495/) (2025)
- [Learning to Lead Themselves: Agentic AI in MAS using MARL](https://mlsanity.com/paper/2510.00022/) (2025)
- [Multi-Agent Environments for Vehicle Routing Problems](https://mlsanity.com/paper/2411.14411/) (2024)
- [Dynamic Sight Range Selection in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.12811/) (2025)
- [Hierarchical Reinforcement Learning with Low-Level MPC for Multi-Agent Control](https://mlsanity.com/paper/2509.15799/) (2025)
- [An effective control of large systems of active particles: An application to evacuation problem](https://mlsanity.com/paper/2509.19972/) (2025)
- [StarCraft+: Benchmarking Multi-agent Algorithms in Adversary Paradigm](https://mlsanity.com/paper/2512.16444/) (2025)
- [MARLEM: A Multi-Agent Reinforcement Learning Simulation Framework for Implicit Cooperation in Decentralized Local Energy Markets](https://mlsanity.com/paper/2602.16063/) (2026)
- [NePPO: Near-Potential Policy Optimization for General-Sum Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2603.06977/) (2026)
- [ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2606.30072/) (2026)
- [Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMs](https://mlsanity.com/paper/2607.08193/) (2026)

## Abstract (arXiv)

A significant challenge facing researchers in the area of multi-agent reinforcement learning (MARL) pertains to the identification of a library that can offer fast and compatible development for multi-agent tasks and algorithm combinations, while obviating the need to consider compatibility issues. In this paper, we present MARLlib, a library designed to address the aforementioned challenge by leveraging three key mechanisms: 1) a standardized multi-agent environment wrapper, 2) an agent-level algorithm implementation, and 3) a flexible policy mapping strategy. By utilizing these mechanisms, MARLlib can effectively disentangle the intertwined nature of the multi-agent task and the learning process of the algorithm, with the ability to automatically alter the training strategy based on the current task's attributes. The MARLlib library's source code is publicly accessible on GitHub: urlhttps://github.com/Replicable-MARL/MARLlib.