# Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization

arXiv: https://arxiv.org/abs/2006.00587 · HTML: https://mlsanity.com/paper/2006.00587/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2020.

## Built on by (most cited first, 26 of 26)

- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare](https://mlsanity.com/paper/2305.01738/) (2023)
- [Learning Long-Term Reward Redistribution via Randomized Return Decomposition](https://mlsanity.com/paper/2111.13485/) (2021)
- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackers](https://mlsanity.com/paper/2305.05909/) (2023)
- [Multi-agent Continual Coordination via Progressive Task Contextualization](https://mlsanity.com/paper/2305.13937/) (2023)
- [Semantically Aligned Task Decomposition in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2305.10865/) (2023)
- [Understanding Value Decomposition Algorithms in Deep Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2202.04868/) (2022)
- [Communication-Robust Multi-Agent Learning by Adaptable Auxiliary Multi-Agent Adversary Generation](https://mlsanity.com/paper/2305.05116/) (2023)
- [Towards Global Optimality in Cooperative MARL with the Transformation And Distillation Framework](https://mlsanity.com/paper/2207.11143/) (2022)
- [On Stateful Value Factorization in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2408.15381/) (2024)
- [Robust Multi-agent Communication via Multi-view Message Certification](https://mlsanity.com/paper/2305.13936/) (2023)
- [Efficient Communication via Self-supervised Information Aggregation for Online and Offline Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2302.09605/) (2023)
- [DISCRET: Synthesizing Faithful Explanations For Treatment Effect Estimation](https://mlsanity.com/paper/2406.00611/) (2024)
- [Focusing Influence Mechanism for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2506.19417/) (2025)
- [A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.02430/) (2023)
- [CuDA2: An approach for Incorporating Traitor Agents into Cooperative Multi-Agent Systems](https://mlsanity.com/paper/2406.17425/) (2024)
- [Tackling Uncertainties in Multi-Agent Reinforcement Learning through Integration of Agent Termination Dynamics](https://mlsanity.com/paper/2501.12061/) (2025)
- [B3C: A Minimalist Approach to Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2501.18138/) (2025)
- [Fixing Incomplete Value Function Decomposition for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2505.10484/) (2025)
- [Partially Decentralized Multi-Agent Q-Learning via Digital Cousins for Wireless Networks](https://mlsanity.com/paper/2503.05970/) (2025)
- [Multi-agent Markov Entanglement](https://mlsanity.com/paper/2506.02385/) (2025)
- [A Learning Framework For Cooperative Collision Avoidance of UAV Swarms Leveraging Domain Knowledge](https://mlsanity.com/paper/2507.10913/) (2025)
- [Structured Cooperative Multi-Agent Reinforcement Learning: a Bayesian Network Perspective](https://mlsanity.com/paper/2510.09937/) (2025)
- [DeepForgeSeal: Latent Space-Driven Semi-Fragile Watermarking for Deepfake Detection Using Adversarial Reinforcement Learning](https://mlsanity.com/paper/2511.04949/) (2025)
- [A Self-Evolving Default Action for Cooperative Tasks with Continuous Action Space](https://mlsanity.com/paper/2607.18597/) (2026)

## Abstract (arXiv)

Value factorization is a popular and promising approach to scaling up multi-agent reinforcement learning in cooperative settings, which balances the learning scalability and the representational capacity of value functions. However, the theoretical understanding of such methods is limited. In this paper, we formalize a multi-agent fitted Q-iteration framework for analyzing factorized multi-agent Q-learning. Based on this framework, we investigate linear value factorization and reveal that multi-agent Q-learning with this simple decomposition implicitly realizes a powerful counterfactual credit assignment, but may not converge in some settings. Through further analysis, we find that on-policy training or richer joint value function classes can improve its local or global convergence properties, respectively. Finally, to support our theoretical implications in practical realization, we conduct an empirical analysis of state-of-the-art deep multi-agent Q-learning algorithms on didactic examples and a broad set of StarCraft II unit micromanagement tasks.