# Toward Risk-based Optimistic Exploration for Cooperative Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2303.01768 · HTML: https://mlsanity.com/paper/2303.01768/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 3 Mar 2023.

## Builds on (most cited first, 24 of 24)

- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [A Distributional Perspective on Reinforcement Learning](https://mlsanity.com/paper/1707.06887/) (2017)
- [QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1803.11485/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [The StarCraft Multi-Agent Challenge](https://mlsanity.com/paper/1902.04043/) (2019)
- [Value-Decomposition Networks For Cooperative Multi-Agent Learning](https://mlsanity.com/paper/1706.05296/) (2017)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1905.05408/) (2019)
- [Implicit Quantile Networks for Distributional Reinforcement Learning](https://mlsanity.com/paper/1806.06923/) (2018)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [MAVEN: Multi-Agent Variational Exploration](https://mlsanity.com/paper/1910.07483/) (2019)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Deep Coordination Graphs](https://mlsanity.com/paper/1910.00091/) (2019)
- [Fully Parameterized Quantile Function for Distributional Reinforcement Learning](https://mlsanity.com/paper/1911.02140/) (2019)
- [Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy](https://mlsanity.com/paper/1911.01546/) (2019)
- [Distributional Reinforcement Learning for Efficient Exploration](https://mlsanity.com/paper/1905.06125/) (2019)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [Information-Directed Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.07544/) (2018)
- [On Bonus-Based Exploration Methods in the Arcade Learning Environment](https://mlsanity.com/paper/2109.11052/) (2021)
- [RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents](https://mlsanity.com/paper/2102.08159/) (2021)
- [DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning](https://mlsanity.com/paper/2102.07936/) (2021)
- [Non-decreasing Quantile Function Network with Efficient Exploration for Distributional Reinforcement Learning](https://mlsanity.com/paper/2105.06696/) (2021)

## Built on by (most cited first, 1 of 1)

- [A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning](https://mlsanity.com/paper/2401.02325/) (2024)

## Abstract (arXiv)

The multi-agent setting is intricate and unpredictable since the behaviors of multiple agents influence one another. To address this environmental uncertainty, distributional reinforcement learning algorithms that incorporate uncertainty via distributional output have been integrated with multi-agent reinforcement learning (MARL) methods, achieving state-of-the-art performance. However, distributional MARL algorithms still rely on the traditional ε-greedy, which does not take cooperative strategy into account. In this paper, we present a risk-based exploration that leads to collaboratively optimistic behavior by shifting the sampling region of distribution. Initially, we take expectations from the upper quantiles of state-action values for exploration, which are optimistic actions, and gradually shift the sampling region of quantiles to the full distribution for exploitation. By ensuring that each agent is exposed to the same level of risk, we can force them to take cooperatively optimistic actions. Our method shows remarkable performance in multi-agent settings requiring cooperative exploration based on quantile regression appropriately controlling the level of risk.