# Generalization through Diversity: Improving Unsupervised Environment Design

arXiv: https://arxiv.org/abs/2301.08025 · HTML: https://mlsanity.com/paper/2301.08025/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 41 references that are in the map.
- First posted on arXiv 19 Jan 2023.

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.05265/) (2021)
- [Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments](https://mlsanity.com/paper/1910.07224/) (2019)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [Replay-Guided Adversarial Environment Design](https://mlsanity.com/paper/2110.02439/) (2021)
- [A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning](https://mlsanity.com/paper/1806.07937/) (2018)
- [Diversity-Driven Exploration Strategy for Deep Reinforcement Learning](https://mlsanity.com/paper/1802.04564/) (2018)
- [Supervision via Competition: Robot Adversaries for Learning Tasks](https://mlsanity.com/paper/1610.01685/) (2016)

## Built on by (most cited first, 9 of 9)

- [Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination](https://mlsanity.com/paper/2504.12714/) (2025)
- [The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Design](https://mlsanity.com/paper/2406.17949/) (2024)
- [SPRINQL: Sub-optimal Demonstrations driven Offline Imitation Learning](https://mlsanity.com/paper/2402.13147/) (2024)
- [Transferable Curricula through Difficulty Conditioned Generators](https://mlsanity.com/paper/2306.13028/) (2023)
- [Unsupervised Partner Design Enables Robust Ad-hoc Teamwork](https://mlsanity.com/paper/2508.06336/) (2025)
- [Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learning](https://mlsanity.com/paper/2608.03571/) (2026)
- [Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs](https://mlsanity.com/paper/2511.12706/) (2025)
- [TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design](https://mlsanity.com/paper/2506.19997/) (2025)
- [Active Curriculum Refinement for Reinforcement Learning](https://mlsanity.com/paper/2608.26469/) (2026)

## Abstract (arXiv)

Agent decision making using Reinforcement Learning (RL) heavily relies on either a model or simulator of the environment (e.g., moving in an 8x8 maze with three rooms, playing Chess on an 8x8 board). Due to this dependence, small changes in the environment (e.g., positions of obstacles in the maze, size of the board) can severely affect the effectiveness of the policy learned by the agent. To that end, existing work has proposed training RL agents on an adaptive curriculum of environments (generated automatically) to improve performance on out-of-distribution (OOD) test scenarios. Specifically, existing research has employed the potential for the agent to learn in an environment (captured using Generalized Advantage Estimation, GAE) as the key factor to select the next environment(s) to train the agent. However, such a mechanism can select similar environments (with a high potential to learn) thereby making agent training redundant on all but one of those environments. To that end, we provide a principled approach to adaptively identify diverse environments based on a novel distance measure relevant to environment design. We empirically demonstrate the versatility and effectiveness of our method in comparison to multiple leading approaches for unsupervised environment design on three distinct benchmark problems used in literature.