# Curiosity-Driven Exploration via Latent Bayesian Surprise

arXiv: https://arxiv.org/abs/2104.07495 · HTML: https://mlsanity.com/paper/2104.07495/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 58 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 30 of 35)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.04717/) (2016)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Noisy Networks for Exploration](https://mlsanity.com/paper/1706.10295/) (2017)
- [Self-Supervised Exploration via Disagreement](https://mlsanity.com/paper/1906.04161/) (2019)
- [Count-Based Exploration with Neural Density Models](https://mlsanity.com/paper/1703.01310/) (2017)
- [Randomized Prior Functions for Deep Reinforcement Learning](https://mlsanity.com/paper/1806.03335/) (2018)
- [Never Give Up: Learning Directed Exploration Strategies](https://mlsanity.com/paper/2002.06038/) (2020)
- [Reinforcement Learning with Prototypical Representations](https://mlsanity.com/paper/2102.11271/) (2021)
- [RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments](https://mlsanity.com/paper/2002.12292/) (2020)
- [State Entropy Maximization with Random Encoders for Efficient Exploration](https://mlsanity.com/paper/2102.09430/) (2021)
- [Model-Based Active Exploration](https://mlsanity.com/paper/1810.12162/) (2018)
- [Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/1703.01732/) (2017)
- [Data-efficient Deep Reinforcement Learning for Dexterous Manipulation](https://mlsanity.com/paper/1704.03073/) (2017)

## Built on by (most cited first, 22 of 22)

- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Choreographer: Learning and Adapting Skills in Imagination](https://mlsanity.com/paper/2211.13350/) (2022)
- [Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels](https://mlsanity.com/paper/2209.12016/) (2022)
- [Reward Shaping for Happier Autonomous Cyber Security Agents](https://mlsanity.com/paper/2310.13565/) (2023)
- [FOCUS: Object-Centric World Models for Robotics Manipulation](https://mlsanity.com/paper/2307.02427/) (2023)
- [PEAC: Unsupervised Pre-training for Cross-Embodiment Reinforcement Learning](https://mlsanity.com/paper/2405.14073/) (2024)
- [EUCLID: Towards Efficient Unsupervised Reinforcement Learning with Multi-choice Dynamics Model](https://mlsanity.com/paper/2210.00498/) (2022)
- [Dr. Strategy: Model-Based Generalist Agents with Strategic Dreaming](https://mlsanity.com/paper/2402.18866/) (2024)
- [Individual Contributions as Intrinsic Exploration Scaffolds for Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2405.18110/) (2024)
- [Wonder Wins Ways: Curiosity-Driven Exploration through Multi-Agent Contextual Calibration](https://mlsanity.com/paper/2509.20648/) (2025)
- [Beyond Noisy-TVs: Noise-Robust Exploration Via Learning Progress Monitoring](https://mlsanity.com/paper/2509.25438/) (2025)
- [Bayesian inference for data-efficient, explainable, and safe robotic motion planning: A review](https://mlsanity.com/paper/2307.08024/) (2023)
- [Adaptive Teaching in Heterogeneous Agents: Balancing Surprise in Sparse Reward Scenarios](https://mlsanity.com/paper/2405.14199/) (2024)
- [Thinking Outside the (Gray) Box: A Context-Based Score for Assessing Value and Originality in Neural Text Generation](https://mlsanity.com/paper/2502.13207/) (2025)
- [Combining Bayesian Inference and Reinforcement Learning for Agent Decision Making: A Review](https://mlsanity.com/paper/2505.07911/) (2025)
- [Action-Adaptive Continual Learning: Enabling Policy Generalization under Dynamic Action Spaces](https://mlsanity.com/paper/2506.05702/) (2025)
- [Curiosity Driven Knowledge Retrieval for Mobile Agents](https://mlsanity.com/paper/2601.19306/) (2026)
- [Task-Aware Exploration via a Predictive Bisimulation Metric](https://mlsanity.com/paper/2602.18724/) (2026)
- [Strangeness-driven Exploration in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2212.13448/) (2022)
- [Self-Emotion-Mediated Exploration in Artificial Intelligence Mirrors: Findings from Cognitive Psychology](https://mlsanity.com/paper/2302.06615/) (2023)
- [Learning to Navigate from Scratch using World Models and Curiosity: the Good, the Bad, and the Ugly](https://mlsanity.com/paper/2308.15852/) (2023)
- [CIG: Exploration via Conditional Information Gain](https://mlsanity.com/paper/2605.20878/) (2026)

## Abstract (arXiv)

The human intrinsic desire to pursue knowledge, also known as curiosity, is considered essential in the process of skill acquisition. With the aid of artificial curiosity, we could equip current techniques for control, such as Reinforcement Learning, with more natural exploration capabilities. A promising approach in this respect has consisted of using Bayesian surprise on model parameters, i.e. a metric for the difference between prior and posterior beliefs, to favour exploration. In this contribution, we propose to apply Bayesian surprise in a latent space representing the agent's current understanding of the dynamics of the system, drastically reducing the computational costs. We extensively evaluate our method by measuring the agent's performance in terms of environment exploration, for continuous tasks, and looking at the game scores achieved, for video games. Our model is computationally cheap and compares positively with current state-of-the-art methods on several problems. We also investigate the effects caused by stochasticity in the environment, which is often a failure case for curiosity-driven agents. In this regime, the results suggest that our approach is resilient to stochastic transitions.