# Multi-Agent Reinforcement Learning with Multi-Step Generative Models

arXiv: https://arxiv.org/abs/1901.10251 · HTML: https://mlsanity.com/paper/1901.10251/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2019.

## Built on by (most cited first, 18 of 18)

- [PLAS: Latent Action Space for Offline Reinforcement Learning](https://mlsanity.com/paper/2011.07213/) (2020)
- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [Scalable Multi-Agent Model-Based Reinforcement Learning](https://mlsanity.com/paper/2205.15023/) (2022)
- [Model-based Multi-agent Policy Optimization with Adaptive Opponent-wise Rollouts](https://mlsanity.com/paper/2105.03363/) (2021)
- [Model-Based Opponent Modeling](https://mlsanity.com/paper/2108.01843/) (2021)
- [Model-based Multi-agent Reinforcement Learning: Recent Progress and Prospects](https://mlsanity.com/paper/2203.10603/) (2022)
- [MAMBPO: Sample-efficient multi-robot reinforcement learning using learned world models](https://mlsanity.com/paper/2103.03662/) (2021)
- [Centralized Model and Exploration Policy for Multi-Agent RL](https://mlsanity.com/paper/2107.06434/) (2021)
- [Multi-Agent Reinforcement Learning with Shared Resources for Inventory Management](https://mlsanity.com/paper/2212.07684/) (2022)
- [MABL: Bi-Level Latent-Variable World Model for Sample-Efficient Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2304.06011/) (2023)
- [Co-Learning Empirical Games and World Models](https://mlsanity.com/paper/2305.14223/) (2023)
- [Agent-Fence: Mapping Security Vulnerabilities Across Deep Research Agents](https://mlsanity.com/paper/2602.07652/) (2026)
- [Ego-centric Learning of Communicative World Models for Autonomous Driving](https://mlsanity.com/paper/2506.08149/) (2025)
- [Training and Evaluation of Deep Policies using Reinforcement Learning and Generative Models](https://mlsanity.com/paper/2204.08573/) (2022)
- [Hierarchical Strategies for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.07397/) (2022)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspective](https://mlsanity.com/paper/2507.09495/) (2025)
- [Reaching Consensus in Cooperative Multi-Agent Reinforcement Learning with Goal Imagination](https://mlsanity.com/paper/2403.03172/) (2024)

## Abstract (arXiv)

We consider model-based reinforcement learning (MBRL) in 2-agent, high-fidelity continuous control problems -- an important domain for robots interacting with other agents in the same workspace. For non-trivial dynamical systems, MBRL typically suffers from accumulating errors. Several recent studies have addressed this problem by learning latent variable models for trajectory segments and optimizing over behavior in the latent space. In this work, we investigate whether this approach can be extended to 2-agent competitive and cooperative settings. The fundamental challenge is how to learn models that capture interactions between agents, yet are disentangled to allow for optimization of each agent behavior separately. We propose such models based on a disentangled variational auto-encoder, and demonstrate our approach on a simulated 2-robot manipulation task, where one robot can either help or distract the other. We show that our approach has better sample efficiency than a strong model-free RL baseline, and can learn both cooperative and adversarial behavior from the same data.