# Qatten: A General Framework for Cooperative Multiagent Reinforcement Learning

arXiv: https://arxiv.org/abs/2002.03939 · HTML: https://mlsanity.com/paper/2002.03939/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Feb 2020.

## Built on by (most cited first, 30 of 81)

- [Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2102.03479/) (2021)
- [An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2409.03052/) (2024)
- [Shapley Counterfactual Credits for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00285/) (2021)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [A Review of Cooperation in Multi-agent Learning](https://mlsanity.com/paper/2312.05162/) (2023)
- [Bridging Evolutionary Algorithms and Reinforcement Learning: A Comprehensive Survey on Hybrid Algorithms](https://mlsanity.com/paper/2401.11963/) (2024)
- [Tesseract: Tensorised Actors for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2106.00136/) (2021)
- [RiskQ: Risk-sensitive Multi-Agent Reinforcement Learning Value Factorization](https://mlsanity.com/paper/2311.01753/) (2023)
- [Regularized Softmax Deep Multi-Agent Q-Learning](https://mlsanity.com/paper/2103.11883/) (2021)
- [Maximum Entropy Heterogeneous-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.10715/) (2023)
- [Divergence-Regularized Multi-Agent Actor-Critic](https://mlsanity.com/paper/2110.00304/) (2021)
- [LDSA: Learning Dynamic Subtask Assignment in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2205.02561/) (2022)
- [Cooperative Multi-Agent Transfer Learning with Level-Adaptive Credit Assignment](https://mlsanity.com/paper/2106.00517/) (2021)
- [Value Functions Factorization with Latent State Information Sharing in Decentralized Multi-Agent Policy Gradients](https://mlsanity.com/paper/2201.01247/) (2022)
- [A Survey on Reinforcement Learning Methods in Character Animation](https://mlsanity.com/paper/2203.04735/) (2022)
- [Learning from Good Trajectories in Offline Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2211.15612/) (2022)
- [Learning Cyber Defence Tactics from Scratch with Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2310.05939/) (2023)
- [From Explicit Communication to Tacit Cooperation:A Novel Paradigm for Cooperative MARL](https://mlsanity.com/paper/2304.14656/) (2023)
- [Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2104.06655/) (2021)
- [Sable: a Performant, Efficient and Scalable Sequence Model for MARL](https://mlsanity.com/paper/2410.01706/) (2024)
- [FoX: Formation-aware exploration in multi-agent reinforcement learning](https://mlsanity.com/paper/2308.11272/) (2023)
- [Efficient Multi-agent Reinforcement Learning by Planning](https://mlsanity.com/paper/2405.11778/) (2024)
- [Learning Generalizable Skills from Offline Multi-Task Data for Multi-Agent Cooperation](https://mlsanity.com/paper/2503.21200/) (2025)
- [CTDS: Centralized Teacher with Decentralized Student for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2203.08412/) (2022)
- [Reinforcement Learning in Factored Action Spaces using Tensor Decompositions](https://mlsanity.com/paper/2110.14538/) (2021)
- [Grounded Answers for Multi-agent Decision-making Problem through Generative World Model](https://mlsanity.com/paper/2410.02664/) (2024)
- [QLLM: Do We Really Need a Mixing Network for Credit Assignment in Multi-Agent Reinforcement Learning?](https://mlsanity.com/paper/2504.12961/) (2025)
- [Learning to Share in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2112.08702/) (2021)
- [Learning to Transfer Role Assignment Across Team Sizes](https://mlsanity.com/paper/2204.12937/) (2022)

## Abstract (arXiv)

In many real-world tasks, multiple agents must learn to coordinate with each other given their private observations and limited communication ability. Deep multiagent reinforcement learning (Deep-MARL) algorithms have shown superior performance in such challenging settings. One representative class of work is multiagent value decomposition, which decomposes the global shared multiagent Q-value Qtot into individual Q-values Qi to guide individuals' behaviors, i.e. VDN imposing an additive formation and QMIX adopting a monotonic assumption using an implicit mixing method. However, most of the previous efforts impose certain assumptions between Qtot and Qi and lack theoretical groundings. Besides, they do not explicitly consider the agent-level impact of individuals to the whole system when transforming individual Qis into Qtot. In this paper, we theoretically derive a general formula of Qtot in terms of Qi, based on which we can naturally implement a multi-head attention formation to approximate Qtot, resulting in not only a refined representation of Qtot with an agent-level attention mechanism, but also a tractable maximization algorithm of decentralized policies. Extensive experiments demonstrate that our method outperforms state-of-the-art MARL methods on the widely adopted StarCraft benchmark across different scenarios, and attention analysis is further conducted with valuable insights.