# QR-MIX: Distributional Value Function Factorisation for Cooperative Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2009.04197 · HTML: https://mlsanity.com/paper/2009.04197/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2020.

## Built on by (most cited first, 5 of 5)

- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [Centralised rehearsal of decentralised cooperation: Multi-agent reinforcement learning for the scalable coordination of residential energy flexibility](https://mlsanity.com/paper/2305.18875/) (2023)
- [MCMARL: Parameterizing Value Function via Mixture of Categorical Distributions for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2202.10134/) (2022)
- [Decision-making with Speculative Opponent Models](https://mlsanity.com/paper/2211.11940/) (2022)
- [A Unified Framework for Factorizing Distributional Value Functions for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2306.02430/) (2023)

## Abstract (arXiv)

In Cooperative Multi-Agent Reinforcement Learning (MARL) and under the setting of Centralized Training with Decentralized Execution (CTDE), agents observe and interact with their environment locally and independently. With local observation and random sampling, the randomness in rewards and observations leads to randomness in long-term returns. Existing methods such as Value Decomposition Network (VDN) and QMIX estimate the value of long-term returns as a scalar that does not contain the information of randomness. Our proposed model QR-MIX introduces quantile regression, modeling joint state-action values as a distribution, combining QMIX with Implicit Quantile Network (IQN). However, the monotonicity in QMIX limits the expression of joint state-action value distribution and may lead to incorrect estimation results in non-monotonic cases. Therefore, we proposed a flexible loss function to approximate the monotonicity found in QMIX. Our model is not only more tolerant of the randomness of returns, but also more tolerant of the randomness of monotonic constraints. The experimental results demonstrate that QR-MIX outperforms the previous state-of-the-art method QMIX in the StarCraft Multi-Agent Challenge (SMAC) environment.