# MAVEN: Multi-Agent Variational Exploration

arXiv: https://arxiv.org/abs/1910.07483 · HTML: https://mlsanity.com/paper/1910.07483/

## Key facts

- Cited by 185 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 30 of 185)

- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Multi-Agent Reinforcement Learning is a Sequence Modeling Problem](https://mlsanity.com/paper/2205.14953/) (2022)
- [Collaborating with Humans without Human Data](https://mlsanity.com/paper/2110.08176/) (2021)
- [SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.07489/) (2022)
- [Celebrating Diversity in Shared Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.02195/) (2021)
- [Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.03479/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.04402/) (2021)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers](https://mlsanity.com/paper/2101.08001/) (2021)
- [Learning more skills through optimistic exploration](https://mlsanity.com/paper/2107.14226/) (2021)
- [Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization](https://mlsanity.com/paper/2103.04564/) (2021)
- [MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer](https://mlsanity.com/paper/2206.10607/) (2022)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition](https://mlsanity.com/paper/2105.08692/) (2021)
- [A Survey on Self-play Methods in Reinforcement Learning](https://mlsanity.com/paper/2408.01072/) (2024)
- [Unbiased Asymmetric Reinforcement Learning under Partial Observability](https://mlsanity.com/paper/2105.11674/) (2021)
- [Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00136/) (2021)
- [PMIC: Improving Multi-Agent Reinforcement Learning with Progressive Mutual Information Collaboration](https://mlsanity.com/paper/2203.08553/) (2022)
- [Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization](https://mlsanity.com/paper/2204.02246/) (2022)
- [Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions](https://mlsanity.com/paper/2209.07899/) (2022)
- [Regularized Softmax Deep Multi-Agent Q-Learning](https://mlsanity.com/paper/2103.11883/) (2021)
- [Interpretation of Emergent Communication in Heterogeneous Collaborative Embodied Agents](https://mlsanity.com/paper/2110.05769/) (2021)
- [HAVEN: Hierarchical Cooperative Multi-Agent Reinforcement Learning with Dual Coordination Mechanism](https://mlsanity.com/paper/2110.07246/) (2021)
- [Generalization in Cooperative Multi-Agent Systems](https://mlsanity.com/paper/2202.00104/) (2022)
- [More Centralized Training, Still Decentralized Execution: Multi-Agent Conditional Policy Factorization](https://mlsanity.com/paper/2209.12681/) (2022)
- [Contrastive Identity-Aware Learning for Multi-Agent Value Decomposition](https://mlsanity.com/paper/2211.12712/) (2022)

## Abstract (arXiv)

Centralised training with decentralised execution is an important setting for cooperative deep multi-agent reinforcement learning due to communication constraints during execution and computational tractability in training. In this paper, we analyse value-based methods that are known to have superior performance in complex environments [43]. We specifically focus on QMIX [40], the current state-of-the-art in this domain. We show that the representational constraints on the joint action-values introduced by QMIX and similar methods lead to provably poor exploration and suboptimality. Furthermore, we propose a novel approach called MAVEN that hybridises value and policy-based methods by introducing a latent space for hierarchical control. The value-based agents condition their behaviour on the shared latent variable controlled by a hierarchical policy. This allows MAVEN to achieve committed, temporally extended exploration, which is key to solving complex multi-agent tasks. Our experimental results show that MAVEN achieves significant performance improvements on the challenging SMAC domain [43].