mlsanityData to 7 Oct 2026

Replay-Guided Adversarial Environment Design

arXiv 2110.02439 ↗First posted 6 Oct 2021cs.LG cs.AIcited by 108

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning (RL) agents may successfully generalize to new settings if trained on an appropriately diverse set of environment and task configurations. Unsupervised Environment Design (UED) is a promising self-supervised RL paradigm, wherein the free parameters of an underspecified environment are automatically adapted during training to the agent's capabilities, leading to the emergence of diverse training environments. Here, we cast Prioritized Level Replay (PLR), an empirically successful but theoretically unmotivated method that selectively samples randomly-generated training levels, as UED. We argue that by curating completely random levels, PLR, too, can generate novel and complex levels for effective training. This insight reveals a natural class of UED methods we call Dual Curriculum Design (DCD). Crucially, DCD includes both PLR and a popular UED algorithm, PAIRED, as special cases and inherits similar theoretical guarantees. This connection allows us to develop novel theory for PLR, providing a version with a robustness guarantee at Nash equilibria. Furthermore, our theory suggests a highly counterintuitive improvement to PLR: by stopping the agent from updating its policy on uncurated levels (training on less data), we can improve the convergence to Nash equilibria. Indeed, our experiments confirm that our new method, PLRperp, obtains better results on a suite of out-of-distribution, zero-shot transfer tasks, in addition to demonstrating that PLRperp improves the performance of PAIRED, from which it inherited its theoretical framework.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Builds on 28 of its 57 references that are in the map. First posted on arXiv 6 Oct 2021.

Builds on

28 of 57 references.

References · 28

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2016OpenAI Gymcited by 1,684
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2018Learning Dexterous In-Hand Manipulationcited by 643
  8. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  9. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  10. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  11. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  12. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
16 more
  1. 2017Teacher-Student Curriculum Learningcited by 197
  2. 2020Prioritized Level Replaycited by 190
  3. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  4. 2019Active Domain Randomizationcited by 161
  5. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  6. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  7. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  8. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  9. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  10. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  11. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  12. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  13. 2019Increasing Generality in Machine Learning through Procedural Content Generationcited by 58
  14. 2020Neuroevolution of Self-Interpretable Agentscited by 48
  15. 2019Unsupervised Curricula for Visual Meta-Reinforcement Learningcited by 33
  16. 2020Adaptive Procedural Task Generation for Hard-Exploration Problemscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 14; 2023: 26; 2024: 25; 2025: 26; 2026: 16.

108 papers.

Sort:

By area:

  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  2. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  3. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  4. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  5. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  6. 2023CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalizationcited by 88
  7. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  8. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  9. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  10. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  11. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  12. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 30
38 more
  1. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  2. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  3. 2025Training a Generally Curious Agentcited by 29
  4. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  5. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 23
  6. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  7. 2022Bayesian Generational Population-Based Trainingcited by 20
  8. 2022General Intelligence Requires Rethinking Explorationcited by 20
  9. 2022Learning Controllable 3D Level Generatorscited by 19
  10. 2022CLUTR: Curriculum Learning via Unsupervised Task Representation Learningcited by 19
  11. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 19
  12. 2024Word2World: Generating Stories and Worlds through Large Language Modelscited by 18
  13. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  14. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  15. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  16. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  17. 2024Acquiring Diverse Skills using Curriculum Reinforcement Learning with Mixture of Expertscited by 14
  18. 2024Eurekaverse: Environment Curriculum Generation via Large Language Modelscited by 14
  19. 2023Reward-Free Curricula for Training Robust World Modelscited by 13
  20. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  21. 2023Train Hard, Fight Easy: Robust Meta Reinforcement Learningcited by 11
  22. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
  23. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  24. 2025Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environmentscited by 11
  25. 2024JaxUED: A simple and useable UED library in Jaxcited by 10
  26. 2025Bootstrapping Task Spaces for Self-Improvementcited by 10
  27. 2022Learning Robust Real-Time Cultural Transmission without Human Datacited by 9
  28. 2023Generalization through Diversity: Improving Unsupervised Environment Designcited by 9
  29. 2023minimax: Efficient Baselines for Autocurricula in JAXcited by 9
  30. 2024Adversarial Environment Design via Regret-Guided Diffusion Modelscited by 9
  31. 2025An Optimisation Framework for Unsupervised Environment Designcited by 9
  32. 2023Surrogate Assisted Generation of Human-Robot Interaction Scenarioscited by 8
  33. 2024Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learningcited by 8
  34. 2025MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spacescited by 7
  35. 2023Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernelcited by 6
  36. 2023On the Benefit of Optimal Transport for Curriculum Reinforcement Learningcited by 6
  37. 2024Proximal Curriculum with Task Correlations for Deep Reinforcement Learningcited by 6
  38. 2024Adaptive Diffusion Terrain Generator for Autonomous Uneven Terrain Navigationcited by 6

50 most cited of 108 All 108 (JSON)