# An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey

arXiv: https://arxiv.org/abs/2209.08890 · HTML: https://mlsanity.com/paper/2209.08890/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 109 of its 217 references that are in the map.
- First posted on arXiv 19 Sep 2022.

## Builds on (most cited first, 30 of 109)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Weight Uncertainty in Neural Networks](https://mlsanity.com/paper/1505.05424/) (2015)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)

## Built on by (most cited first, 28 of 28)

- [MaxInfoRL: Boosting exploration in reinforcement learning through information gain maximization](https://mlsanity.com/paper/2412.12098/) (2024)
- [Why AI systems don't learn and what to do about it: Lessons on autonomous learning from cognitive science](https://mlsanity.com/paper/2603.15381/) (2026)
- [RLeXplore: Accelerating Research in Intrinsically-Motivated Reinforcement Learning](https://mlsanity.com/paper/2405.19548/) (2024)
- [Hieros: Hierarchical Imagination on Structured State Space Sequence World Models](https://mlsanity.com/paper/2310.05167/) (2023)
- [The impact of intrinsic rewards on exploration in Reinforcement Learning](https://mlsanity.com/paper/2501.11533/) (2025)
- [Intrinsically-Motivated Humans and Agents in Open-World Exploration](https://mlsanity.com/paper/2503.23631/) (2025)
- [SOMBRL: Scalable and Optimistic Model-Based RL](https://mlsanity.com/paper/2511.20066/) (2025)
- [Beyond Optimism: Exploration With Partially Observable Rewards](https://mlsanity.com/paper/2406.13909/) (2024)
- [Constrained Skill Discovery: Quadruped Locomotion with Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2410.07877/) (2024)
- [From Curiosity to Competence: How World Models Interact with the Dynamics of Exploration](https://mlsanity.com/paper/2507.08210/) (2025)
- [How Should We Meta-Learn Reinforcement Learning Algorithms?](https://mlsanity.com/paper/2507.17668/) (2025)
- [BAMDP Shaping: a Unified Framework for Intrinsic Motivation and Reward Shaping](https://mlsanity.com/paper/2409.05358/) (2024)
- [Intrinsic Motivation in Model-based Reinforcement Learning: A Brief Review](https://mlsanity.com/paper/2301.10067/) (2023)
- [Image-Based Deep Reinforcement Learning with Intrinsically Motivated Stimuli: On the Execution of Complex Robotic Tasks](https://mlsanity.com/paper/2407.21338/) (2024)
- [Diversity Progress for Goal Selection in Discriminability-Motivated RL](https://mlsanity.com/paper/2411.01521/) (2024)
- [Self-supervised network distillation: an effective approach to exploration in sparse reward environments](https://mlsanity.com/paper/2302.11563/) (2023)
- [InDRiVE: Intrinsic Disagreement based Reinforcement for Vehicle Exploration through Curiosity Driven Generalized World Model](https://mlsanity.com/paper/2503.05573/) (2025)
- [Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models](https://mlsanity.com/paper/2505.07815/) (2025)
- [VendiRL: A Framework for Self-Supervised Reinforcement Learning of Diversely Diverse Skills](https://mlsanity.com/paper/2509.02930/) (2025)
- [A general Markov decision process formalism for action-state entropy-regularized reward maximization](https://mlsanity.com/paper/2302.01098/) (2023)
- [Emergence of Goal-Directed Behaviors via Active Inference with Self-Prior](https://mlsanity.com/paper/2504.11075/) (2025)
- [Towards a Formal Theory of the Need for Competence via Computational Intrinsic Motivation](https://mlsanity.com/paper/2502.07423/) (2025)
- [Can In-Context Learning Support Intrinsic Curiosity?](https://mlsanity.com/paper/2606.19476/) (2026)
- [Principled Direction-Free Intrinsic Motivation through Model-Free Epistemic Free-Energy Estimators](https://mlsanity.com/paper/2607.16858/) (2026)
- [Policy Complexity, Reaction Time, and Bounded Rationality in Reinforcement Learning](https://mlsanity.com/paper/2609.28737/) (2026)
- [Tactile Curiosity Drives Robot Interaction](https://mlsanity.com/paper/2609.40134/) (2026)
- [Reward Inflation: A Healthy Stimulus for Reinforcement Learning](https://mlsanity.com/paper/2610.02545/) (2026)
- [Mastering Atari 2600 Games with Discovered Options](https://mlsanity.com/paper/2610.03604/) (2026)

## Abstract (arXiv)

The reinforcement learning (RL) research area is very active, with an important number of new contributions; especially considering the emergent field of deep RL (DRL). However a number of scientific and technical challenges still need to be resolved, amongst which we can mention the ability to abstract actions or the difficulty to explore the environment in sparse-reward settings which can be addressed by intrinsic motivation (IM). We propose to survey these research works through a new taxonomy based on information theory: we computationally revisit the notions of surprise, novelty and skill learning. This allows us to identify advantages and disadvantages of methods and exhibit current outlooks of research. Our analysis suggests that novelty and surprise can assist the building of a hierarchy of transferable skills that further abstracts the environment and makes the exploration process more robust.