# Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation

arXiv: https://arxiv.org/abs/1806.10729 · HTML: https://mlsanity.com/paper/1806.10729/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jun 2018.

## Built on by (most cited first, 30 of 81)

- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Domain Generalization with MixStyle](https://mlsanity.com/paper/2104.02008/) (2021)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability](https://mlsanity.com/paper/2107.06277/) (2021)
- [Efficient Reinforcement Finetuning via Adaptive Curriculum Learning](https://mlsanity.com/paper/2504.05520/) (2025)
- [Video as the New Language for Real-World Decision Making](https://mlsanity.com/paper/2402.17139/) (2024)
- [Decoupling Value and Policy for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2102.10330/) (2021)
- [Open-Endedness is Essential for Artificial Superhuman Intelligence](https://mlsanity.com/paper/2406.04268/) (2024)
- [MixStyle Neural Networks for Domain Generalization and Adaptation](https://mlsanity.com/paper/2107.02053/) (2021)
- [Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay](https://mlsanity.com/paper/2506.05316/) (2025)
- [Adversarially Guided Actor-Critic](https://mlsanity.com/paper/2102.04376/) (2021)
- [DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training](https://mlsanity.com/paper/2504.09710/) (2025)
- [SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies](https://mlsanity.com/paper/2106.09678/) (2021)
- [Learning Controllable Content Generators](https://mlsanity.com/paper/2105.02993/) (2021)
- [Deep Surrogate Assisted Generation of Environments](https://mlsanity.com/paper/2206.04199/) (2022)
- [Adversarial Reinforcement Learning for Procedural Content Generation](https://mlsanity.com/paper/2103.04847/) (2021)
- [Procedural Generalization by Planning with Self-Supervised World Models](https://mlsanity.com/paper/2111.01587/) (2021)
- [On the Importance of Exploration for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2306.05483/) (2023)
- [The Generalization Gap in Offline Reinforcement Learning](https://mlsanity.com/paper/2312.05742/) (2023)
- [Quantifying the effects of environment and population diversity in multi-agent reinforcement learning](https://mlsanity.com/paper/2102.08370/) (2021)
- [Benchmarking Reinforcement Learning Techniques for Autonomous Navigation](https://mlsanity.com/paper/2210.04839/) (2022)
- [TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL](https://mlsanity.com/paper/2103.09815/) (2021)
- [Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds](https://mlsanity.com/paper/2210.13417/) (2022)
- [Multi-Robot Coordination and Layout Design for Automated Warehousing](https://mlsanity.com/paper/2305.06436/) (2023)
- [DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraft](https://mlsanity.com/paper/2404.15538/) (2024)
- [Horizon Generalization in Reinforcement Learning](https://mlsanity.com/paper/2501.02709/) (2025)
- [It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generation](https://mlsanity.com/paper/2202.10608/) (2022)

## Abstract (arXiv)

Deep reinforcement learning (RL) has shown impressive results in a variety of domains, learning directly from high-dimensional sensory streams. However, when neural networks are trained in a fixed environment, such as a single level in a video game, they will usually overfit and fail to generalize to new levels. When RL models overfit, even slight modifications to the environment can result in poor agent performance. This paper explores how procedurally generated levels during training can increase generality. We show that for some games procedural level generation enables generalization to new levels within the same distribution. Additionally, it is possible to achieve better performance with less data by manipulating the difficulty of the levels in response to the performance of the agent. The generality of the learned behaviors is also evaluated on a set of human-designed levels. The results suggest that the ability to generalize to human-designed levels highly depends on the design of the level generators. We apply dimensionality reduction and clustering techniques to visualize the generators' distributions of levels and analyze to what degree they can produce levels similar to those designed by a human.