# SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments

arXiv: https://arxiv.org/abs/1912.05510 · HTML: https://mlsanity.com/paper/1912.05510/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2019.

## Built on by (most cited first, 9 of 9)

- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning](https://mlsanity.com/paper/2110.12997/) (2021)
- [On efficient computation in active inference](https://mlsanity.com/paper/2307.00504/) (2023)
- [CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning](https://mlsanity.com/paper/2306.13412/) (2023)
- [Human Cognition in Machines: A Unified Perspective of World Models](https://mlsanity.com/paper/2604.16592/) (2026)
- [Of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents](https://mlsanity.com/paper/2505.12204/) (2025)
- [Adapting Surprise Minimizing Reinforcement Learning Techniques for Transactive Control](https://mlsanity.com/paper/2111.06025/) (2021)
- [Reinforcement Learning Textbook](https://mlsanity.com/paper/2201.09746/) (2022)
- [Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization](https://mlsanity.com/paper/2211.06236/) (2022)

## Abstract (arXiv)

Every living organism struggles against disruptive environmental forces to carve out and maintain an orderly niche. We propose that such a struggle to achieve and preserve order might offer a principle for the emergence of useful behaviors in artificial agents. We formalize this idea into an unsupervised reinforcement learning method called surprise minimizing reinforcement learning (SMiRL). SMiRL alternates between learning a density model to evaluate the surprise of a stimulus, and improving the policy to seek more predictable stimuli. The policy seeks out stable and repeatable situations that counteract the environment's prevailing sources of entropy. This might include avoiding other hostile agents, or finding a stable, balanced pose for a bipedal robot in the face of disturbance forces. We demonstrate that our surprise minimizing agents can successfully play Tetris, Doom, control a humanoid to avoid falls, and navigate to escape enemies in a maze without any task-specific reward supervision. We further show that SMiRL can be used together with standard task rewards to accelerate reward-driven learning.