# Reinforcement Learning Generalization with Surprise Minimization

arXiv: https://arxiv.org/abs/2004.12399 · HTML: https://mlsanity.com/paper/2004.12399/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Apr 2020.

## Built on by (most cited first, 10 of 10)

- [A Survey of Zero-shot Generalisation in Deep Reinforcement Learning](https://mlsanity.com/paper/2111.09794/) (2021)
- [Decoupling Value and Policy for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2102.10330/) (2021)
- [On the Importance of Exploration for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2306.05483/) (2023)
- [The Generalization Gap in Offline Reinforcement Learning](https://mlsanity.com/paper/2312.05742/) (2023)
- [From Psychological Curiosity to Artificial Curiosity: Curiosity-Driven Learning in Artificial Intelligence Tasks](https://mlsanity.com/paper/2201.08300/) (2022)
- [Review of Metrics to Measure the Stability, Robustness and Resilience of Reinforcement Learning](https://mlsanity.com/paper/2203.12048/) (2022)
- [Evolving Pareto-Optimal Actor-Critic Algorithms for Generalizability and Stability](https://mlsanity.com/paper/2204.04292/) (2022)
- [Improving generalization to new environments and removing catastrophic forgetting in Reinforcement Learning by using an eco-system of agents](https://mlsanity.com/paper/2204.06550/) (2022)
- [Improving generalization in reinforcement learning through forked agents](https://mlsanity.com/paper/2212.06451/) (2022)
- [Leveraging weights signals -- Predicting and improving generalizability in reinforcement learning](https://mlsanity.com/paper/2511.20234/) (2025)

## Abstract (arXiv)

Generalization remains a challenging problem for deep reinforcement learning algorithms, which are often trained and tested on the same set of deterministic game environments. When test environments are unseen and perturbed but the nature of the task remains the same, generalization gaps can arise. In this work, we propose and evaluate a surprise minimizing agent on a generalization benchmark to show an additional reward learned from a simple density model can show robustness in procedurally generated game environments that provide constant source of entropy and stochasticity.