mlsanityData to 14 Oct 2026

Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design

arXiv 2012.02096 ↗First posted 3 Dec 2020cs.LG cs.AI cs.MAcited by 275

Not on the map. Why?

Cited by 60 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A wide range of reinforcement learning (RL) problems - including robustness, transfer learning, unsupervised RL, and emergent complexity - require specifying a distribution of tasks or environments in which a policy will be trained. However, creating a useful distribution of environments is error prone, and takes a significant amount of developer time and effort. We propose Unsupervised Environment Design (UED) as an alternative paradigm, where developers provide environments with unknown parameters, and these parameters are used to automatically produce a distribution over valid, solvable environments. Existing approaches to automatically generating environments suffer from common failure modes: domain randomization cannot generate structure or adapt the difficulty of the environment to the agent's learning progress, and minimax adversarial training leads to worst-case environments that are often unsolvable. To generate structured, solvable environments for our protagonist agent, we introduce a second, antagonist agent that is allied with the environment-generating adversary. The adversary is motivated to generate environments which maximize regret, defined as the difference between the protagonist and antagonist agent's return. We call our technique Protagonist Antagonist Induced Regret Environment Design (PAIRED). Our experiments demonstrate that PAIRED produces a natural curriculum of increasingly complex environments, and PAIRED agents achieve higher zero-shot transfer performance when tested in highly novel environments.

Key facts. Cited by 275 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 21; 2022: 45; 2023: 39; 2024: 47; 2025: 57; 2026: 66.

275 papers.

Sort:

By area:

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  3. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  4. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  5. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  6. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  7. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  8. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  9. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  10. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  11. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  12. 2025Self-Evolving Curriculum for LLM Reasoningcited by 112
38 more
  1. 2021Replay-Guided Adversarial Environment Designcited by 111
  2. 2025Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Modelscited by 109
  3. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 97
  4. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
  5. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 80
  6. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 70
  7. 2023OMNI: Open-endedness via Models of human Notions of Interestingnesscited by 70
  8. 2021Self-Supervised Bug Detection and Repaircited by 59
  9. 2022Exploration via Elliptical Episodic Bonusescited by 51
  10. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 46
  11. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 45
  12. 2021Learning Controllable Content Generatorscited by 40
  13. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  14. 2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38
  15. 2021Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agentscited by 36
  16. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 36
  17. 2021Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraftcited by 35
  18. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  19. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  20. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 31
  21. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 31
  22. 2025Training a Generally Curious Agentcited by 31
  23. 2022Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Trainingcited by 28
  24. 2022Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptationcited by 28
  25. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 25
  26. 2025Learning to Reason at the Frontier of Learnabilitycited by 23
  27. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  28. 2022Bayesian Generational Population-Based Trainingcited by 20
  29. 2022CLUTR: Curriculum Learning via Unsupervised Task Representation Learningcited by 20
  30. 2022General Intelligence Requires Rethinking Explorationcited by 20
  31. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 20
  32. 2024AdaSociety: An Adaptive Environment with Social Structures for Multi-Agent Decision-Makingcited by 20
  33. 2024Automating the Search for Artificial Life with Foundation Modelscited by 20
  34. 2022Learning Controllable 3D Level Generatorscited by 19
  35. 2022Benchmarking Reinforcement Learning Techniques for Autonomous Navigationcited by 19
  36. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 18
  37. 2025BoxingGym: Benchmarking Progress in Automated Experimental Design and Model Discoverycited by 18
  38. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 18

50 most cited of 275 All 275 (JSON)