# Adventurer: Exploration with BiGAN for Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/2503.18612 · HTML: https://mlsanity.com/paper/2503.18612/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 30 of its 51 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery](https://mlsanity.com/paper/1703.05921/) (2017)
- [Adversarial Feature Learning](https://mlsanity.com/paper/1605.09782/) (2016)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Improving Unsupervised Defect Segmentation by Applying Structural Similarity to Autoencoders](https://mlsanity.com/paper/1807.02011/) (2018)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Deep Learning for Anomaly Detection: A Review](https://mlsanity.com/paper/2007.02500/) (2020)
- [IPO: Interior-point Policy Optimization under Constraints](https://mlsanity.com/paper/1910.09615/) (2019)
- [Efficient GAN-Based Anomaly Detection](https://mlsanity.com/paper/1802.06222/) (2018)
- [Adversarially Learned Anomaly Detection](https://mlsanity.com/paper/1812.02288/) (2018)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [EX2: Exploration with Exemplar Models for Deep Reinforcement Learning](https://mlsanity.com/paper/1703.01260/) (2017)
- [On Bonus-Based Exploration Methods in the Arcade Learning Environment](https://mlsanity.com/paper/2109.11052/) (2021)
- [A Survey on GANs for Anomaly Detection](https://mlsanity.com/paper/1906.11632/) (2019)
- [Curiosity-Driven Experience Prioritization via Density Estimation](https://mlsanity.com/paper/1902.08039/) (2019)
- [A survey on Variational Autoencoders from a GreenAI perspective](https://mlsanity.com/paper/2103.01071/) (2021)
- [Focus on Impact: Indoor Exploration with Intrinsic Motivation](https://mlsanity.com/paper/2109.08521/) (2021)
- [CLARA: A Constrained Reinforcement Learning Based Resource Allocation Framework for Network Slicing](https://mlsanity.com/paper/2111.08397/) (2021)
- [Generative Adversarial Exploration for Reinforcement Learning](https://mlsanity.com/paper/2201.11685/) (2022)

## Abstract (arXiv)

Recent developments in deep reinforcement learning have been very successful in learning complex, previously intractable problems. Sample efficiency and local optimality, however, remain significant challenges. To address these challenges, novelty-driven exploration strategies have emerged and shown promising potential. Unfortunately, no single algorithm outperforms all others in all tasks and most of them struggle with tasks with high-dimensional and complex observations. In this work, we propose Adventurer, a novelty-driven exploration algorithm that is based on Bidirectional Generative Adversarial Networks (BiGAN), where BiGAN is trained to estimate state novelty. Intuitively, a generator that has been trained on the distribution of visited states should only be able to generate a state coming from the distribution of visited states. As a result, novel states using the generator to reconstruct input states from certain latent representations would lead to larger reconstruction errors. We show that BiGAN performs well in estimating state novelty for complex observations. This novelty estimation method can be combined with intrinsic-reward-based exploration. Our empirical results show that Adventurer produces competitive results on a range of popular benchmark tasks, including continuous robotic manipulation tasks (e.g. Mujoco robotics) and high-dimensional image-based tasks (e.g. Atari games).