# Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2003.08839 · HTML: https://mlsanity.com/paper/2003.08839/

## Key facts

- Cited by 485 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2020.

## Built on by (most cited first, 30 of 485)

- [MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory](https://mlsanity.com/paper/2601.03192/) (2026)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [MindAgent: Emergent Gaming Interaction](https://mlsanity.com/paper/2309.09971/) (2023)
- [Scaling Multi-Agent Reinforcement Learning with Selective Parameter Sharing](https://mlsanity.com/paper/2102.07475/) (2021)
- [MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning](https://mlsanity.com/paper/2502.18439/) (2025)
- [Simplifying Deep Temporal Difference Learning](https://mlsanity.com/paper/2407.04811/) (2024)
- [MADiff: Offline Multi-agent Learning with Diffusion Models](https://mlsanity.com/paper/2305.17330/) (2023)
- [LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions](https://mlsanity.com/paper/2405.11106/) (2024)
- [JaxMARL: Multi-Agent RL Environments and Algorithms in JAX](https://mlsanity.com/paper/2311.10090/) (2023)
- [Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectification](https://mlsanity.com/paper/2111.11188/) (2021)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind](https://mlsanity.com/paper/2111.09189/) (2021)
- [An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.03052/) (2024)
- [Learning Multi-Agent Communication from Graph Modeling Perspective](https://mlsanity.com/paper/2405.08550/) (2024)
- [Towards Efficient LLM Grounding for Embodied Multi-Agent Collaboration](https://mlsanity.com/paper/2405.14314/) (2024)
- [Deep Multiagent Reinforcement Learning: Challenges and Directions](https://mlsanity.com/paper/2106.15691/) (2021)
- [Improving Retrieval-Augmented Generation through Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2501.15228/) (2025)
- [Controlling Large Language Model-based Agents for Large-Scale Decision-Making: An Actor-Critic Approach](https://mlsanity.com/paper/2311.13884/) (2023)
- [Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2309.12696/) (2023)
- [Multi-Agent Coordination across Diverse Applications: A Survey](https://mlsanity.com/paper/2502.14743/) (2025)
- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [XDO: A Double Oracle Algorithm for Extensive-Form Games](https://mlsanity.com/paper/2103.06426/) (2021)
- [MASER: Multi-Agent Reinforcement Learning with Subgoals Generated from Experience Replay Buffer](https://mlsanity.com/paper/2206.10607/) (2022)
- [OmniDrones: An Efficient and Flexible Platform for Reinforcement Learning in Drone Control](https://mlsanity.com/paper/2309.12825/) (2023)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [Multi-Agent Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2408.09675/) (2024)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition](https://mlsanity.com/paper/2105.08692/) (2021)
- [Towards Distraction-Robust Active Visual Tracking](https://mlsanity.com/paper/2106.10110/) (2021)
- [DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning](https://mlsanity.com/paper/2102.07936/) (2021)

## Abstract (arXiv)

In many real-world settings, a team of agents must coordinate its behaviour while acting in a decentralised fashion. At the same time, it is often possible to train the agents in a centralised fashion where global state information is available and communication constraints are lifted. Learning joint action-values conditioned on extra state information is an attractive way to exploit centralised learning, but the best strategy for then extracting decentralised policies is unclear. Our solution is QMIX, a novel value-based method that can train decentralised policies in a centralised end-to-end fashion. QMIX employs a mixing network that estimates joint action-values as a monotonic combination of per-agent values. We structurally enforce that the joint-action value is monotonic in the per-agent values, through the use of non-negative weights in the mixing network, which guarantees consistency between the centralised and decentralised policies. To evaluate the performance of QMIX, we propose the StarCraft Multi-Agent Challenge (SMAC) as a new benchmark for deep multi-agent reinforcement learning. We evaluate QMIX on a challenging set of SMAC scenarios and show that it significantly outperforms existing multi-agent reinforcement learning methods.