mlsanityData to 14 Oct 2026

Stabilizing Unsupervised Environment Design with a Learned Adversary

arXiv 2308.10797 ↗First posted 21 Aug 2023cs.LG cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

A key challenge in training generally-capable agents is the design of training tasks that facilitate broad generalization and robustness to environment variations. This challenge motivates the problem setting of Unsupervised Environment Design (UED), whereby a student agent trains on an adaptive distribution of tasks proposed by a teacher agent. A pioneering approach for UED is PAIRED, which uses reinforcement learning (RL) to train a teacher policy to design tasks from scratch, making it possible to directly generate tasks that are adapted to the agent's current capabilities. Despite its strong theoretical backing, PAIRED suffers from a variety of challenges that hinder its practical performance. Thus, state-of-the-art methods currently rely on curation and mutation rather than generation of new tasks. In this work, we investigate several key shortcomings of PAIRED and propose solutions for each shortcoming. As a result, we make it possible for PAIRED to match or exceed state-of-the-art methods, producing robust agents in several established challenging procedurally-generated environments, including a partially-observed maze navigation task and a continuous-control car racing environment. We believe this work motivates a renewed emphasis on UED methods based on learned models that directly generate challenging environments, potentially unlocking more open-ended RL training and, as a result, more general agents.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

36 of 57 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2013Intriguing properties of neural networkscited by 5,155
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  6. 2017Robust Adversarial Reinforcement Learningcited by 451
  7. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  8. 2019Adversarial Policies: Attacking Deep Reinforcement Learningcited by 214
  9. 2020Prioritized Level Replaycited by 198
  10. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  11. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  12. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
24 more
  1. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 166
  2. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  3. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 138
  4. 2021Goal Misgeneralization in Deep Reinforcement Learningcited by 123
  5. 2021Replay-Guided Adversarial Environment Designcited by 111
  6. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 106
  7. 2017Delving into adversarial attacks on deep policiescited by 104
  8. 2019Observational Overfitting in Reinforcement Learningcited by 91
  9. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 87
  10. 2021Off-Belief Learningcited by 75
  11. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 72
  12. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  13. 2020Neuroevolution of Self-Interpretable Agentscited by 49
  14. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  15. 2021Illuminating Diverse Neural Cellular Automata for Level Generationcited by 35
  16. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  17. 2018Measuring and Characterizing Generalization in Deep Reinforcement Learningcited by 20
  18. 2016Supervision via Competition: Robot Adversaries for Learning Taskscited by 16
  19. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  20. 2020Co-generation of game levels and game-playing agentscited by 14
  21. 2022It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generationcited by 14
  22. 2020Robust Domain Randomised Reinforcement Learning through Peer-to-Peer Distillationcited by 10
  23. 2022Mutation Models: Learning to Generate Levels by Imitating Evolutioncited by 9
  24. 2022Transfer Dynamics in Emergent Evolutionary Curriculacited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 3; 2026: 2.

17 papers.

Sort:

By area:

  1. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 31
  2. 2024AdaSociety: An Adaptive Environment with Social Structures for Multi-Agent Decision-Makingcited by 20
  3. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 18
  4. 2024Eurekaverse: Environment Curriculum Generation via Large Language Modelscited by 16
  5. 2025Cross-environment Cooperation Enables Zero-shot Multi-agent Coordinationcited by 14
  6. 2024JaxUED: A simple and useable UED library in Jaxcited by 11
  7. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  8. 2023minimax: Efficient Baselines for Autocurricula in JAXcited by 10
  9. 2024Adversarial Environment Design via Regret-Guided Diffusion Modelscited by 10
  10. 2024Multi-Agent Diagnostics for Robustness via Illuminated Diversitycited by 4
  11. 2025Improving Environment Novelty Quantification for Effective Unsupervised Environment Designcited by 4
  12. 2024Scenario-Based Curriculum Generation for Multi-Agent Autonomous Drivingcited by 3
5 more
  1. 2024Autoverse: An Evolvable Game Language for Learning Robust Embodied Agentscited by 3
  2. 2024Syllabus: Portable Curricula for Reinforcement Learning Agentscited by 2
  3. 2025TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Designcited by 1
  4. 2026Improving Regret Approximation for Unsupervised Dynamic Environment Generationcited by 1
  5. 2026Active Curriculum Refinement for Reinforcement Learningcited by 1