# VIME: Variational Information Maximizing Exploration

arXiv: https://arxiv.org/abs/1605.09674 · HTML: https://mlsanity.com/paper/1605.09674/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2016.

## Built on by (most cited first, 16 of 16)

- [APS: Active Pretraining with Successor Features](https://mlsanity.com/paper/2108.13956/) (2021)
- [KL Convergence Guarantees for Score diffusion models under minimal data assumptions](https://mlsanity.com/paper/2308.12240/) (2023)
- [World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry](https://mlsanity.com/paper/2604.01985/) (2026)
- [Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration](https://mlsanity.com/paper/2108.06038/) (2021)
- [Off-Policy RL Algorithms Can be Sample-Efficient for Continuous Control via Sample Multiple Reuse](https://mlsanity.com/paper/2305.18443/) (2023)
- [k-Means Maximum Entropy Exploration](https://mlsanity.com/paper/2205.15623/) (2022)
- [Touch-based Curiosity for Sparse-Reward Tasks](https://mlsanity.com/paper/2104.00442/) (2021)
- [Data-Efficient Task Generalization via Probabilistic Model-based Meta Reinforcement Learning](https://mlsanity.com/paper/2311.07558/) (2023)
- [Active Sensing with Predictive Coding and Uncertainty Minimization](https://mlsanity.com/paper/2307.00668/) (2023)
- [Normality-Guided Distributional Reinforcement Learning for Continuous Control](https://mlsanity.com/paper/2208.13125/) (2022)
- [Using Curiosity for an Even Representation of Tasks in Continual Offline Reinforcement Learning](https://mlsanity.com/paper/2312.03177/) (2023)
- [The Geometry of Empowerment](https://mlsanity.com/paper/2610.07796/) (2026)
- [Self-Supervised Exploration via Temporal Inconsistency in Reinforcement Learning](https://mlsanity.com/paper/2208.11361/) (2022)
- [Strangeness-driven Exploration in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.13448/) (2022)
- [Curiosity-Driven Reinforcement Learning based Low-Level Flight Control](https://mlsanity.com/paper/2307.15724/) (2023)
- [AdaKP: Online Adaptive Knowledge-Point Selection for Reasoning-Oriented Reinforcement Learning](https://mlsanity.com/paper/2607.24833/) (2026)

## Abstract (arXiv)

Scalable and effective exploration remains a key challenge in reinforcement learning (RL). While there are methods with optimality guarantees in the setting of discrete state and action spaces, these methods cannot be applied in high-dimensional deep RL scenarios. As such, most contemporary RL relies on simple heuristics such as epsilon-greedy exploration or adding Gaussian noise to the controls. This paper introduces Variational Information Maximizing Exploration (VIME), an exploration strategy based on maximization of information gain about the agent's belief of environment dynamics. We propose a practical implementation, using variational inference in Bayesian neural networks which efficiently handles continuous state and action spaces. VIME modifies the MDP reward function, and can be applied with several different underlying RL algorithms. We demonstrate that VIME achieves significantly better performance compared to heuristic exploration methods across a variety of continuous control tasks and algorithms, including tasks with very sparse rewards.