Improving Regret Approximation for Unsupervised Dynamic Environment Generation
Not on the map. Why?
Abstract (arXiv)
Unsupervised Environment Design (UED) seeks to automatically generate training curricula for reinforcement learning (RL) agents, with the goal of improving generalisation and zero-shot performance. However, designing effective curricula remains a difficult problem, particularly in settings where small subsets of environment parameterisations result in significant increases in the complexity of the required policy. Current methods struggle with a difficult credit assignment problem and rely on regret approximations that fail to identify challenging levels, both of which are compounded as the size of the environment grows. We propose Dynamic Environment Generation for UED (DEGen) to enable a denser level generator reward signal, reducing the difficulty of credit assignment and allowing for UED to scale to larger environment sizes. We also introduce a new regret approximation, Maximised Negative Advantage (MNA), as a significantly improved metric to optimise for, that better identifies more challenging levels. We show empirically that MNA outperforms current regret approximations and when combined with DEGen, consistently outperforms existing methods, especially as the size of the environment grows. We have made all our code available here: https://github.com/HarryMJMead/Dynamic-Environment-Generation-for-UED.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 48 of its 64 references that are in the map. First posted on arXiv 21 Jan 2026.
Builds on
48 of 64 references.
References · 48
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
- 2018World Modelscited by 1,444
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
- 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
- 2024Genie: Generative Interactive Environmentscited by 706
- 2019Solving Rubik's Cube with a Robot Handcited by 699
- 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
- 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
- 2023Learning Interactive Real-World Simulatorscited by 440
- 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
36 more
- 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
- 2018Quantifying Generalization in Reinforcement Learningcited by 345
- 2022Transformers are Sample-Efficient World Modelscited by 317
- 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
- 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
- 2017Teacher-Student Curriculum Learningcited by 197
- 2020Prioritized Level Replaycited by 190
- 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
- 2022Evolving Curricula with Regret-Based Environment Designcited by 166
- 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
- 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
- 2025Training Agents Inside of Scalable World Modelscited by 122
- 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
- 2021Replay-Guided Adversarial Environment Designcited by 108
- 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
- 2020PCGRL: Procedural Content Generation via Reinforcement Learningcited by 90
- 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
- 2019Increasing Generality in Machine Learning through Procedural Content Generationcited by 58
- 2021Learning Controllable Content Generatorscited by 40
- 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
- 2020Adaptive Procedural Task Generation for Hard-Exploration Problemscited by 20
- 2025Learning to Reason at the Frontier of Learnabilitycited by 20
- 2022CLUTR: Curriculum Learning via Unsupervised Task Representation Learningcited by 19
- 2023Proximal Curriculum for Reinforcement Learning Agentscited by 19
- 2018Imitation Learning with Concurrent Actions in 3D Gamescited by 18
- 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
- 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
- 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
- 2023Reward-Free Curricula for Training Robust World Modelscited by 13
- 2024JaxUED: A simple and useable UED library in Jaxcited by 10
- 2024Adversarial Environment Design via Regret-Guided Diffusion Modelscited by 9
- 2025An Optimisation Framework for Unsupervised Environment Designcited by 9
- 202110 Years of the PCG workshop: Past and Future Trendscited by 6
- 2024Proximal Curriculum with Task Correlations for Deep Reinforcement Learningcited by 6
- 2025Imagined Autocurriculacited by 4
- 2019Procedural Generation of Initial States of Sokobancited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.