# MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models

arXiv: https://arxiv.org/abs/2103.03662 · HTML: https://mlsanity.com/paper/2103.03662/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 11 of its 17 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 11 of 11)

- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Actor-Attention-Critic for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1810.02912/) (2018)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Deep Decentralized Multi-task Multi-Agent Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/1703.06182/) (2017)
- [Regularization for Deep Learning: A Taxonomy](https://mlsanity.com/paper/1710.10686/) (2017)
- [Multi-Agent Reinforcement Learning with Multi-Step Generative Models](https://mlsanity.com/paper/1901.10251/) (2019)

## Built on by (most cited first, 16 of 16)

- [Model-Based Opponent Modeling](https://mlsanity.com/paper/2108.01843/) (2021)
- [A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem](https://mlsanity.com/paper/2305.17198/) (2023)
- [Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects](https://mlsanity.com/paper/2203.10603/) (2022)
- [Centralized Model and Exploration Policy for Multi-Agent RL](https://mlsanity.com/paper/2107.06434/) (2021)
- [Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models](https://mlsanity.com/paper/2406.15836/) (2024)
- [Efficient Multi-agent Reinforcement Learning by Planning](https://mlsanity.com/paper/2405.11778/) (2024)
- [CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spaces](https://mlsanity.com/paper/2211.15824/) (2022)
- [CoDreamer: Communication-Based Decentralised World Models](https://mlsanity.com/paper/2406.13600/) (2024)
- [Co-Learning Empirical Games and World Models](https://mlsanity.com/paper/2305.14223/) (2023)
- [Nearest-Neighbor-based Collision Avoidance for Quadrotors via Reinforcement Learning](https://mlsanity.com/paper/2104.14912/) (2021)
- [Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.06274/) (2022)
- [Model-Based Decentralized Policy Optimization](https://mlsanity.com/paper/2302.08139/) (2023)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Toward Adaptable Multi-Agent Reinforcement Learning: An Assumption-Aware Review](https://mlsanity.com/paper/2507.10142/) (2025)
- [An Algorithm For Adversary Aware Decentralized Networked MARL](https://mlsanity.com/paper/2305.05573/) (2023)
- [Formal Modelling for Multi-Robot Systems Under Uncertainty](https://mlsanity.com/paper/2305.17018/) (2023)

## Abstract (arXiv)

Multi-robot systems can benefit from reinforcement learning (RL) algorithms that learn behaviours in a small number of trials, a property known as sample efficiency. This research thus investigates the use of learned world models to improve sample efficiency. We present a novel multi-agent model-based RL algorithm: Multi-Agent Model-Based Policy Optimization (MAMBPO), utilizing the Centralized Learning for Decentralized Execution (CLDE) framework. CLDE algorithms allow a group of agents to act in a fully decentralized manner after training. This is a desirable property for many systems comprising of multiple robots. MAMBPO uses a learned world model to improve sample efficiency compared to model-free Multi-Agent Soft Actor-Critic (MASAC). We demonstrate this on two simulated multi-robot tasks, where MAMBPO achieves a similar performance to MASAC, but requires far fewer samples to do so. Through this, we take an important step towards making real-life learning for multi-robot systems possible.