mlsanityData to 7 Oct 2026

Deep Surrogate Assisted Generation of Environments

arXiv 2206.04199 ↗First posted 9 Jun 2022cs.AI cs.LG cs.NEcited by 40

Not on the map. Why?

Abstract (arXiv)

Recent progress in reinforcement learning (RL) has started producing generally capable agents that can solve a distribution of complex environments. These agents are typically tested on fixed, human-authored environments. On the other hand, quality diversity (QD) optimization has been proven to be an effective component of environment generation algorithms, which can generate collections of high-quality environments that are diverse in the resulting agent behaviors. However, these algorithms require potentially expensive simulations of agents on newly generated environments. We propose Deep Surrogate Assisted Generation of Environments (DSAGE), a sample-efficient QD environment generation algorithm that maintains a deep surrogate model for predicting agent behaviors in new environments. Results in two benchmark domains show that DSAGE significantly outperforms existing QD environment generation algorithms in discovering collections of environments that elicit diverse behaviors of a state-of-the-art RL agent and a planning agent. Our source code and videos are available at https://dsagepaper.github.io/.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 61 of its 118 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

61 of 118 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2017Improved Training of Wasserstein GANscited by 2,223
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  8. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  9. 2015Illuminating search spaces by mapping elitescited by 406
  10. 2021Unsolved Problems in ML Safetycited by 291
  11. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  12. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
38 more
  1. 2014Robots that can adapt like animalscited by 203
  2. 2020Prioritized Level Replaycited by 190
  3. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  4. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  5. 2017Procedural Content Generation via Machine Learning (PCGML)cited by 136
  6. 2021Replay-Guided Adversarial Environment Designcited by 108
  7. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  8. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  9. 2020PCGRL: Procedural Content Generation via Reinforcement Learningcited by 90
  10. 2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 80
  11. 2018Scenic: A Language for Scenario Specification and Scene Generationcited by 79
  12. 2019Covariance Matrix Adaptation for the Rapid Illumination of Behavior Spacecited by 78
  13. 2018Evolving Mario Levels in the Latent Space of a Deep Convolutional Generative Adversarial Networkcited by 76
  14. 2020Deep Learning for Procedural Content Generationcited by 73
  15. 2021Differentiable Quality Diversitycited by 71
  16. 2019Increasing Generality in Machine Learning through Procedural Content Generationcited by 58
  17. 2020Illuminating Mario Scenes in the Latent Space of a Generative Adversarial Networkcited by 51
  18. 2019Procedural Content Generation through Quality Diversitycited by 45
  19. 2016Super Mario as a String: Platformer Level Generation Via LSTMscited by 40
  20. 2021Learning Controllable Content Generatorscited by 40
  21. 2019Learning to Combat Compounding-Error in Model-Based Reinforcement Learningcited by 39
  22. 2018Data-Efficient Design Exploration through Surrogate-Assisted Illuminationcited by 38
  23. 2020Scaling MAP-Elites to Deep Neuroevolutioncited by 38
  24. 2021On the Importance of Environments in Human-Robot Coordinationcited by 37
  25. 2022Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learningcited by 37
  26. 2019Bootstrapping Conditional GANs for Video Game Level Generationcited by 35
  27. 2021Illuminating Diverse Neural Cellular Automata for Level Generationcited by 35
  28. 2021Generating and Blending Game Levels via Quality-Diversity in the Latent Space of a Variational Autoencodercited by 27
  29. 2018Talakat: Bullet Hell Generation through Constrained Map-Elitescited by 26
  30. 2021Deep Surrogate Assisted MAP-Elites for Automated Hearthstone Deckbuildingcited by 26
  31. 2020A Quality Diversity Approach to Automatically Generating Human-Robot Interaction Scenarios in Shared Autonomycited by 20
  32. 2021Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolutioncited by 20
  33. 2021Dynamics-Aware Quality-Diversity for Efficient Learning of Skill Repertoirescited by 20
  34. 2022Insights From the NeurIPS 2021 NetHack Challengecited by 20
  35. 2020CPPN2GAN: Combining Compositional Pattern Producing Networks and GANs for Large-scale Pattern Generationcited by 19
  36. 2020Conditional Level Generation and Game Blendingcited by 17
  37. 2018DOOM Level Generation using Generative Adversarial Networkscited by 15
  38. 2020BOP-Elites, a Bayesian Optimisation algorithm for Quality-Diversity searchcited by 15

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 9; 2025: 5; 2026: 9.

40 papers.

Sort:

By area:

  1. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  2. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  3. 2023pyribs: A Bare-Bones Python Library for Quality Diversity Optimizationcited by 29
  4. 2023Evolutionary Reinforcement Learning: A Surveycited by 24
  5. 2022General Intelligence Requires Rethinking Explorationcited by 20
  6. 2023Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learningcited by 20
  7. 2023Generating Behaviorally Diverse Policies with Latent Diffusion Modelscited by 18
  8. 2023Multi-Robot Coordination and Layout Design for Automated Warehousingcited by 17
  9. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  10. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  11. 2023Robust multi-agent coordination via evolutionary generation of auxiliary adversarial attackerscited by 15
  12. 2024Eurekaverse: Environment Curriculum Generation via Large Language Modelscited by 14
28 more
  1. 2022Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealingcited by 10
  2. 2023Arbitrarily Scalable Environment Generators via Neural Cellular Automatacited by 9
  3. 2024Adversarial Environment Design via Regret-Guided Diffusion Modelscited by 9
  4. 2023Surrogate Assisted Generation of Human-Robot Interaction Scenarioscited by 8
  5. 2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 8
  6. 2023Diversity from Human Feedbackcited by 5
  7. 2022Augmentative Topology Agents For Open-Ended Learningcited by 3
  8. 2024Enabling Adaptive Agent Training in Open-Ended Simulators by Targeting Diversitycited by 3
  9. 2025AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimizationcited by 3
  10. 2023Enhancing the Hierarchical Environment Design via Generative Trajectory Modelingcited by 2
  11. 2024Quality-Diversity with Limited Resourcescited by 1
  12. 2024Algorithmic Scenario Generation as Quality Diversity Optimizationcited by 1
  13. 2024QD-MAPPER: A Quality Diversity Framework to Automatically Evaluate Multi-Agent Path Finding Algorithms in Diverse Mapscited by 1
  14. 2024Preference-Conditioned Gradient Variations for Multi-Objective Quality-Diversitycited by 1
  15. 2025Surrogate Fitness Metrics for Interpretable Reinforcement Learningcited by 1
  16. 2025Soft Quality-Diversity Optimizationcited by 1
  17. 2026Discount Model Search for Quality Diversity Optimization in High-Dimensional Measure Spacescited by 1
  18. 2026Optimization of Edge Directions and Weights for Mixed Guidance Graphs in Lifelong Multi-Agent Path Findingcited by 1
  19. 2026Red-Teaming Vision-Language-Action Models via Quality Diversity Prompt Generation for Robust Robot Policiescited by 1
  20. 2026Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigationcited by 1
  21. 2023Evolutionary Machine Learning and Gamescited by 0
  22. 2024Future Research Avenues for Artificial Intelligence in Digital Gaming: An Exploratory Reportcited by 0
  23. 2025SCOUT: A Lightweight Framework for Scenario Coverage Assessment in Autonomous Drivingcited by 0
  24. 2026Quality-Diversity Optimization as Multi-Objective Optimizationcited by 0
  25. 2026Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learningcited by 0
  26. 2026U-Net-Accelerated Quality-Diversity Optimization for Climate-Adaptive Urban Layoutscited by 0
  27. 2026Gimitest: A Comprehensive Tool for Testing Reinforcement Learning Policiescited by 0
  28. 2026Stress-Relief Annealing: Polynomial-Time Simulation-Free Layout Optimization for Automated Warehousescited by 0