mlsanityData to 14 Oct 2026

An Optimisation Framework for Unsupervised Environment Design

arXiv 2505.20659 ↗First posted 27 May 2025cs.LGcited by 10

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

For reinforcement learning agents to be deployed in high-risk settings, they must achieve a high level of robustness to unfamiliar scenarios. One method for improving robustness is unsupervised environment design (UED), a suite of methods aiming to maximise an agent's generalisability across configurations of an environment. In this work, we study UED from an optimisation perspective, providing stronger theoretical guarantees for practical settings than prior work. Whereas previous methods relied on guarantees if they reach convergence, our framework employs a nonconvex-strongly-concave objective for which we provide a provably convergent algorithm in the zero-sum setting. We empirically verify the efficacy of our method, outperforming prior methods in a number of environments with varying difficulties.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 43 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

34 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Adversarial Environment Design via Regret-Guided Diffusion Models25 Oct
  2. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery27 Aug
  3. 2024Refining Minimax Regret for Unsupervised Environment Design19 Feb
  4. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design5 Feb

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  3. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  4. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  5. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  6. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
  7. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 270
  8. 2020Prioritized Level Replaycited by 198
  9. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  10. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  11. 2020A Two-Timescale Framework for Bilevel Optimization: Complexity Analysis and Application to Actor-Criticcited by 144
  12. 2018Optimistic mirror descent in saddle-point problems: Going the extra (gradient) milecited by 136
18 more
  1. 2021Replay-Guided Adversarial Environment Designcited by 111
  2. 2018Last-Iterate Convergence: Zero-Sum Games and Constrained Min-Max Optimizationcited by 98
  3. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 88
  4. 2018Stable Recurrent Modelscited by 66
  5. 2019Stabilizing Generative Adversarial Networks: A Surveycited by 57
  6. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  7. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 45
  8. 2019Is the Policy Gradient a Gradient?cited by 40
  9. 2018Robust Optimization over Multiple Domainscited by 34
  10. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  11. 2020Last iterate convergence in no-regret learning: constrained min-max optimization for convex-concave landscapescited by 21
  12. 2022Accelerated Algorithms for Constrained Nonconvex-Nonconcave Min-Max Optimization and Comonotone Inclusioncited by 20
  13. 2022CLUTR: Curriculum Learning via Unsupervised Task Representation Learningcited by 20
  14. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 20
  15. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  16. 2022TiAda: A Time-scale Adaptive Algorithm for Nonconvex Minimax Optimizationcited by 8
  17. 2024Learning Equilibria in Adversarial Team Markov Games: A Nonconvex-Hidden-Concave Min-Max Optimization Problemcited by 5
  18. 2023Enhancing the Hierarchical Environment Design via Generative Trajectory Modelingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 12
  2. 2025Unsupervised Partner Design Enables Robust Ad-hoc Teamworkcited by 3
  3. 2025Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observabilitycited by 3
  4. 2025Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairscited by 2
  5. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxiescited by 2
  6. 2025Mitigating Goal Misgeneralization via Minimax Regretcited by 1
  7. 2026Improving Regret Approximation for Unsupervised Dynamic Environment Generationcited by 1
  8. 2026Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Designcited by 1
  9. 2026Active Curriculum Refinement for Reinforcement Learningcited by 1
  10. 2026PACE: Parameter Change for Unsupervised Environment Designcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learning
  2. 2025TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design
  3. 2022A Mixture of Surprises for Unsupervised Reinforcement Learning
  4. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  5. 2026Unsupervised Learning of Efficient Exploration: Pre-training Adaptive Policies via Self-Imposed Goals
  6. 2024A Safe Exploration Strategy for Model-free Task Adaptation in Safety-constrained Grid Environments
  7. 2022Environment Design for Inverse Reinforcement Learning
  8. 2024Safe and Robust Reinforcement Learning: Principles and Practice