# Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2302.10825 · HTML: https://mlsanity.com/paper/2302.10825/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 66 references that are in the map.
- First posted on arXiv 21 Feb 2023.

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks](https://mlsanity.com/paper/2006.07869/) (2020)
- [Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO](https://mlsanity.com/paper/2005.12729/) (2020)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Cooperative Exploration for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/2107.11444/) (2021)
- [On Improving Model-Free Algorithms for Decentralized Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2110.05707/) (2021)
- [Message-Dropout: An Efficient Training Method for Multi-Agent Deep Reinforcement Learning](https://mlsanity.com/paper/1902.06527/) (2019)
- [A New Framework for Multi-Agent Reinforcement Learning -- Centralized Training and Exploration with Decentralized Execution via Policy Distillation](https://mlsanity.com/paper/1910.09152/) (2019)
- [Counterfactual Multi-Agent Reinforcement Learning with Graph Convolution Communication](https://mlsanity.com/paper/2004.00470/) (2020)
- [Proximal Policy Optimization via Enhanced Exploration Efficiency](https://mlsanity.com/paper/2011.05525/) (2020)
- [Multi-Agent Reinforcement Learning for Problems with Combined Individual and Team Reward](https://mlsanity.com/paper/2003.10598/) (2020)
- [An Evaluation Study of Intrinsic Motivation Techniques applied to Reinforcement Learning over Hard Exploration Environments](https://mlsanity.com/paper/2205.11184/) (2022)
- [Generative Personas That Behave and Experience Like Humans](https://mlsanity.com/paper/2209.00459/) (2022)
- [Play with Emotion: Affect-Driven Reinforcement Learning](https://mlsanity.com/paper/2208.12622/) (2022)
- [CH-MARL: A Multimodal Benchmark for Cooperative, Heterogeneous Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2208.13626/) (2022)
- [Go-Explore Complex 3D Game Environments for Automated Reachability Testing](https://mlsanity.com/paper/2209.00570/) (2022)
- [Curiosity-Driven Multi-Agent Exploration with Mixed Objectives](https://mlsanity.com/paper/2210.16468/) (2022)
- [First Go, then Post-Explore: the Benefits of Post-Exploration in Intrinsic Motivation](https://mlsanity.com/paper/2212.03251/) (2022)

## Built on by (most cited first, 1 of 1)

- [From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism](https://mlsanity.com/paper/2604.04648/) (2026)

## Abstract (arXiv)

Sparsity of rewards while applying a deep reinforcement learning method negatively affects its sample-efficiency. A viable solution to deal with the sparsity of rewards is to learn via intrinsic motivation which advocates for adding an intrinsic reward to the reward function to encourage the agent to explore the environment and expand the sample space. Though intrinsic motivation methods are widely used to improve data-efficient learning in the reinforcement learning model, they also suffer from the so-called detachment problem. In this article, we discuss the limitations of intrinsic curiosity module in sparse-reward multi-agent reinforcement learning and propose a method called I-Go-Explore that combines the intrinsic curiosity module with the Go-Explore framework to alleviate the detachment problem.