mlsanityData to 7 Oct 2026

Evolving Curricula with Regret-Based Environment Design

arXiv 2203.01302 ↗First posted 2 Mar 2022cs.LGcited by 166

Not on the map. Why?

Abstract (arXiv)

It remains a significant challenge to train generally capable agents with reinforcement learning (RL). A promising avenue for improving the robustness of RL agents is through the use of curricula. One such class of methods frames environment design as a game between a student and a teacher, using regret-based objectives to produce environment instantiations (or levels) at the frontier of the student agent's capabilities. These methods benefit from their generality, with theoretical guarantees at equilibrium, yet they often struggle to find effective levels in challenging design spaces. By contrast, evolutionary approaches seek to incrementally alter environment complexity, resulting in potentially open-ended learning, but often rely on domain-specific heuristics and vast amounts of computational resources. In this paper we propose to harness the power of evolution in a principled, regret-based curriculum. Our approach, which we call Adversarially Compounding Complexity by Editing Levels (ACCEL), seeks to constantly produce levels at the frontier of an agent's capabilities, resulting in curricula that start simple but become increasingly complex. ACCEL maintains the theoretical benefits of prior regret-based methods, while providing significant empirical gains in a diverse set of environments. An interactive version of the paper is available at accelagent.github.io.

Key facts. Cited by 166 papers in the map (data through 7 Oct 2026). Builds on 65 of its 101 references that are in the map. First posted on arXiv 2 Mar 2022.

Builds on

65 of 101 references.

References · 65

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2016OpenAI Gymcited by 1,684
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2019Solving Rubik's Cube with a Robot Handcited by 699
  10. 2018Learning Dexterous In-Hand Manipulationcited by 643
  11. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
38 more
  1. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  2. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  3. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  4. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  5. 2017Teacher-Student Curriculum Learningcited by 197
  6. 2020The NetHack Learning Environmentcited by 196
  7. 2020Prioritized Level Replaycited by 190
  8. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  9. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  10. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  11. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  12. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  13. 2019Active Domain Randomizationcited by 161
  14. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  15. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  16. 2017Procedural Content Generation via Machine Learning (PCGML)cited by 136
  17. 2018Assessing Generalization in Deep Reinforcement Learningcited by 132
  18. 2021Discovering and Achieving Goals via World Modelscited by 131
  19. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  20. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  21. 2021Replay-Guided Adversarial Environment Designcited by 108
  22. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  23. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  24. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101
  25. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  26. 2019Observational Overfitting in Reinforcement Learningcited by 90
  27. 2020PCGRL: Procedural Content Generation via Reinforcement Learningcited by 90
  28. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  29. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  30. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  31. 2020Deep Learning for Procedural Content Generationcited by 73
  32. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  33. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  34. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  35. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  36. 2019Increasing Generality in Machine Learning through Procedural Content Generationcited by 58
  37. 2017Markov Decision Processes with Continuous Side Informationcited by 54
  38. 2019Obstacle Tower: A Generalization Challenge in Vision, Control, and Planningcited by 54

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 33; 2024: 35; 2025: 38; 2026: 40.

166 papers.

Sort:

By area:

  1. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
  2. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  3. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  4. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  5. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  6. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  7. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 102
  8. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  9. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  10. 2025STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Provingcited by 77
  11. 2024EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agentscited by 41
  12. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
38 more
  1. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  2. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  3. 2025Prompt Curriculum Learning for Efficient LLM Post-Trainingcited by 40
  4. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 30
  5. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 30
  6. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  7. 2023The Generalization Gap in Offline Reinforcement Learningcited by 25
  8. 2023The SocialAI School: Insights from Developmental Psychology Towards Artificial Socio-Cultural Agentscited by 24
  9. 2024DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedbackcited by 23
  10. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  11. 2022General Intelligence Requires Rethinking Explorationcited by 20
  12. 2024Automating the Search for Artificial Life with Foundation Modelscited by 20
  13. 2025Learning to Reason at the Frontier of Learnabilitycited by 20
  14. 2022Learning Controllable 3D Level Generatorscited by 19
  15. 2022CLUTR: Curriculum Learning via Unsupervised Task Representation Learningcited by 19
  16. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 19
  17. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  18. 2024DreamCraft: Text-Guided Generation of Functional 3D Environments in Minecraftcited by 17
  19. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 16
  20. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  21. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  22. 2022The Effectiveness of World Models for Continual Reinforcement Learningcited by 15
  23. 2024Eurekaverse: Environment Curriculum Generation via Large Language Modelscited by 14
  24. 2023Reward-Free Curricula for Training Robust World Modelscited by 13
  25. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learningcited by 13
  26. 2022Heterogeneous Multi-agent Zero-Shot Coordination by Coevolutioncited by 12
  27. 2024Robust Fast Adaptation from Adversarially Explicit Task Distribution Generationcited by 12
  28. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
  29. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  30. 2024JaxUED: A simple and useable UED library in Jaxcited by 10
  31. 2025Bootstrapping Task Spaces for Self-Improvementcited by 10
  32. 2023Generalization through Diversity: Improving Unsupervised Environment Designcited by 9
  33. 2023Arbitrarily Scalable Environment Generators via Neural Cellular Automatacited by 9
  34. 2023minimax: Efficient Baselines for Autocurricula in JAXcited by 9
  35. 2024Adversarial Environment Design via Regret-Guided Diffusion Modelscited by 9
  36. 2025An Optimisation Framework for Unsupervised Environment Designcited by 9
  37. 2023Surrogate Assisted Generation of Human-Robot Interaction Scenarioscited by 8
  38. 2026Digital Red Queen: Adversarial Program Evolution in Core War with LLMscited by 7

50 most cited of 166 All 166 (JSON)