# Implicit Generative Modeling for Efficient Exploration

arXiv: https://arxiv.org/abs/1911.08017 · HTML: https://mlsanity.com/paper/1911.08017/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Nov 2019.

## Built on by (most cited first, 6 of 6)

- [Principled Exploration via Optimistic Bootstrapping and Backward Induction](https://mlsanity.com/paper/2105.06022/) (2021)
- [Curiosity-Driven Exploration via Latent Bayesian Surprise](https://mlsanity.com/paper/2104.07495/) (2021)
- [Dynamic Bottleneck for Robust Self-Supervised Exploration](https://mlsanity.com/paper/2110.10735/) (2021)
- [COPlanner: Plan to Roll Out Conservatively but to Explore Optimistically for Model-Based RL](https://mlsanity.com/paper/2310.07220/) (2023)
- [Enhancing Diversity in Bayesian Deep Learning via Hyperspherical Energy Minimization of CKA](https://mlsanity.com/paper/2411.00259/) (2024)
- [SEREN: Knowing When to Explore and When to Exploit](https://mlsanity.com/paper/2205.15064/) (2022)

## Abstract (arXiv)

Efficient exploration remains a challenging problem in reinforcement learning, especially for those tasks where rewards from environments are sparse. A commonly used approach for exploring such environments is to introduce some "intrinsic" reward. In this work, we focus on model uncertainty estimation as an intrinsic reward for efficient exploration. In particular, we introduce an implicit generative modeling approach to estimate a Bayesian uncertainty of the agent's belief of the environment dynamics. Each random draw from our generative model is a neural network that instantiates the dynamic function, hence multiple draws would approximate the posterior, and the variance in the future prediction based on this posterior is used as an intrinsic reward for exploration. We design a training algorithm for our generative model based on the amortized Stein Variational Gradient Descent. In experiments, we compare our implementation with state-of-the-art intrinsic reward-based exploration approaches, including two recent approaches based on an ensemble of dynamic models. In challenging exploration tasks, our implicit generative model consistently outperforms competing approaches regarding data efficiency in exploration.