# Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

arXiv: https://arxiv.org/abs/1509.08731 · HTML: https://mlsanity.com/paper/1509.08731/

## Key facts

- Cited by 144 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Sep 2015.

## Built on by (most cited first, 30 of 144)

- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [APS: Active Pretraining with Successor Features](https://mlsanity.com/paper/2108.13956/) (2021)
- [Invariant Information Bottleneck for Domain Generalization](https://mlsanity.com/paper/2106.06333/) (2021)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Causal Influence Detection for Improving Efficiency in Reinforcement Learning](https://mlsanity.com/paper/2106.03443/) (2021)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [Episodic Multi-agent Reinforcement Learning with Curiosity-Driven Exploration](https://mlsanity.com/paper/2111.11032/) (2021)
- [Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models](https://mlsanity.com/paper/2207.08229/) (2022)
- [MADE: Exploration via Maximizing Deviation from Explored Regions](https://mlsanity.com/paper/2106.10268/) (2021)
- [The Information Geometry of Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2110.02719/) (2021)
- [Adversarial Imitation via Variational Inverse Reinforcement Learning](https://mlsanity.com/paper/1809.06404/) (2018)
- [Curious Exploration via Structured World Models Yields Zero-Shot Object Manipulation](https://mlsanity.com/paper/2206.11403/) (2022)
- [Which Mutual-Information Representation Learning Objectives are Sufficient for Control?](https://mlsanity.com/paper/2106.07278/) (2021)
- [How to Stay Curious while Avoiding Noisy TVs using Aleatoric Uncertainty Estimation](https://mlsanity.com/paper/2102.04399/) (2021)
- [INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL](https://mlsanity.com/paper/2204.08585/) (2022)
- [Task-Agnostic Morphology Evolution](https://mlsanity.com/paper/2102.13100/) (2021)
- [Any-Play: An Intrinsic Augmentation for Zero-Shot Coordination](https://mlsanity.com/paper/2201.12436/) (2022)
- [Learning How to Propagate Messages in Graph Neural Networks](https://mlsanity.com/paper/2310.00697/) (2023)
- [Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning](https://mlsanity.com/paper/2102.13515/) (2021)
- [Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions](https://mlsanity.com/paper/2209.07899/) (2022)
- [Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning](https://mlsanity.com/paper/2412.08021/) (2024)
- [Maximize to Explore: One Objective Function Fusing Estimation, Planning, and Exploration](https://mlsanity.com/paper/2305.18258/) (2023)
- [Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching](https://mlsanity.com/paper/2110.14457/) (2021)
- [Theory of Mind as Intrinsic Motivation for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2307.01158/) (2023)
- [Mutual Information State Intrinsic Control](https://mlsanity.com/paper/2103.08107/) (2021)
- [Active Hierarchical Exploration with Stable Subgoal Representation Learning](https://mlsanity.com/paper/2105.14750/) (2021)
- [Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration](https://mlsanity.com/paper/2108.06038/) (2021)
- [Curious Replay for Model-based Adaptation](https://mlsanity.com/paper/2306.15934/) (2023)

## Abstract (arXiv)

The mutual information is a core statistical quantity that has applications in all areas of machine learning, whether this is in training of density models over multiple data modalities, in maximising the efficiency of noisy transmission channels, or when learning behaviour policies for exploration by artificial agents. Most learning algorithms that involve optimisation of the mutual information rely on the Blahut-Arimoto algorithm --- an enumerative algorithm with exponential complexity that is not suitable for modern machine learning applications. This paper provides a new approach for scalable optimisation of the mutual information by merging techniques from variational inference and deep learning. We develop our approach by focusing on the problem of intrinsically-motivated learning, where the mutual information forms the definition of a well-known internal drive known as empowerment. Using a variational lower bound on the mutual information, combined with convolutional networks for handling visual input streams, we develop a stochastic optimisation algorithm that allows for scalable information maximisation and empowerment-based reasoning directly from pixels to actions.