# Multi-Agent Collaboration via Reward Attribution Decomposition

arXiv: https://arxiv.org/abs/2010.08531 · HTML: https://mlsanity.com/paper/2010.08531/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2020.

## Built on by (most cited first, 17 of 17)

- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition](https://mlsanity.com/paper/2105.08692/) (2021)
- [Collaborative Visual Navigation](https://mlsanity.com/paper/2107.01151/) (2021)
- [Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration](https://mlsanity.com/paper/2108.06038/) (2021)
- [ACE: Cooperative Multi-agent Q-learning with Bidirectional Action-Dependency](https://mlsanity.com/paper/2211.16068/) (2022)
- [Spatial Intention Maps for Multi-Agent Mobile Manipulation](https://mlsanity.com/paper/2103.12710/) (2021)
- [M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality](https://mlsanity.com/paper/2503.02077/) (2025)
- [LINDA: Multi-Agent Local Information Decomposition for Awareness of Teammates](https://mlsanity.com/paper/2109.12508/) (2021)
- [Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.17540/) (2022)
- [S2RL: Do We Really Need to Perceive All States in Deep Multi-Agent Reinforcement Learning?](https://mlsanity.com/paper/2206.11054/) (2022)
- [CORD: Generalizable Cooperation via Role Diversity](https://mlsanity.com/paper/2501.02221/) (2025)
- [RACA: Relation-Aware Credit Assignment for Ad-Hoc Cooperation in Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2206.01207/) (2022)
- [Learning cooperative behaviours in adversarial multi-agent systems](https://mlsanity.com/paper/2302.05528/) (2023)
- [CoMIX: A Multi-agent Reinforcement Learning Training Architecture for Efficient Decentralized Coordination and Independent Decision-Making](https://mlsanity.com/paper/2308.10721/) (2023)
- [Learning Bilateral Team Formation in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2506.20039/) (2025)

## Abstract (arXiv)

Recent advances in multi-agent reinforcement learning (MARL) have achieved super-human performance in games like Quake 3 and Dota 2. Unfortunately, these techniques require orders-of-magnitude more training rounds than humans and don't generalize to new agent configurations even on the same game. In this work, we propose Collaborative Q-learning (CollaQ) that achieves state-of-the-art performance in the StarCraft multi-agent challenge and supports ad hoc team play. We first formulate multi-agent collaboration as a joint optimization on reward assignment and show that each agent has an approximately optimal policy that decomposes into two parts: one part that only relies on the agent's own state, and the other part that is related to states of nearby agents. Following this novel finding, CollaQ decomposes the Q-function of each agent into a self term and an interactive term, with a Multi-Agent Reward Attribution (MARA) loss that regularizes the training. CollaQ is evaluated on various StarCraft maps and shows that it outperforms existing state-of-the-art techniques (i.e., QMIX, QTRAN, and VDN) by improving the win rate by 40% with the same number of samples. In the more challenging ad hoc team play setting (i.e., reweight/add/remove units without re-training or finetuning), CollaQ outperforms previous SoTA by over 30%.