mlsanityData to 7 Oct 2026

Refining Minimax Regret for Unsupervised Environment Design

arXiv 2402.12284 ↗First posted 19 Feb 2024cs.LG cs.AIcited by 16

Not on the map. Why?

Abstract (arXiv)

In unsupervised environment design, reinforcement learning agents are trained on environment configurations (levels) generated by an adversary that maximises some objective. Regret is a commonly used objective that theoretically results in a minimax regret (MMR) policy with desirable robustness guarantees; in particular, the agent's maximum regret is bounded. However, once the agent reaches this regret bound on all levels, the adversary will only sample levels where regret cannot be further reduced. Although there are possible performance improvements to be made outside of these regret-maximising levels, learning stagnates. In this work, we introduce Bayesian level-perfect MMR (BLP), a refinement of the minimax regret objective that overcomes this limitation. We formally show that solving for this objective results in a subset of MMR policies, and that BLP policies act consistently with a Perfect Bayesian policy over all levels. We further introduce an algorithm, ReMiDi, that results in a BLP policy at convergence. We empirically demonstrate that training on levels from a minimax regret adversary causes learning to prematurely stagnate, but that ReMiDi continues learning.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 28 of its 56 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

28 of 56 references.

References · 28

  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  2. 2017Robust Adversarial Reinforcement Learningcited by 439
  3. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  4. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  5. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  6. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  7. 2020Prioritized Level Replaycited by 190
  8. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  9. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  10. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  11. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  12. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
16 more
  1. 2021Replay-Guided Adversarial Environment Designcited by 108
  2. 2022Discovered Policy Optimisationcited by 103
  3. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  4. 2021Off-Belief Learningcited by 72
  5. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 64
  6. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  7. 2023Emergence of Maps in the Memories of Blind Navigation Agentscited by 31
  8. 2023Adversarial Model for Offline Reinforcement Learningcited by 31
  9. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  10. 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 29
  11. 2023On the Foundation of Distributionally Robust Reinforcement Learningcited by 18
  12. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  13. 2022Grounding Aleatoric Uncertainty for Unsupervised Environment Designcited by 16
  14. 2019Distributionally Robust Partially Observable Markov Decision Process with Moment-based Ambiguitycited by 15
  15. 2020Distributionally Robust Markov Decision Processes and their Connection to Risk Measurescited by 12
  16. 2023minimax: Efficient Baselines for Autocurricula in JAXcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 6.

16 papers.

Sort:

By area:

  1. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  2. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  3. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  4. 2025An Optimisation Framework for Unsupervised Environment Designcited by 9
  5. 2026EnvHarness: Awakening Static Worlds for Agent Learningcited by 7
  6. 2025Improving Environment Novelty Quantification for Effective Unsupervised Environment Designcited by 4
  7. 2024Syllabus: Portable Curricula for Reinforcement Learning Agentscited by 2
  8. 2025Unsupervised Partner Design Enables Robust Ad-hoc Teamworkcited by 2
  9. 2025Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairscited by 2
  10. 2025Mitigating Goal Misgeneralization via Minimax Regretcited by 1
  11. 2026Co-Evolution in Agentic Systems: Toward Self-Directed Evolution Beyond Human Designcited by 1
  12. 2026Active Curriculum Refinement for Reinforcement Learningcited by 1
4 more
  1. 2025Task Scheduling & Forgetting in Multi-Task Reinforcement Learningcited by 0
  2. 2026D'ej`aQ: Open-Ended Evolution of Diverse, Learnable and Verifiable Problemscited by 0
  3. 2026Predicting Task Difficulty Without Rolloutscited by 0
  4. 2026Efficient Exploration Is Enoughcited by 0