mlsanityData to 7 Oct 2026

MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning

arXiv 2303.03376 ↗First posted 6 Mar 2023cs.LG cs.MAcited by 30

Not on the map. Why?

Abstract (arXiv)

Open-ended learning methods that automatically generate a curriculum of increasingly challenging tasks serve as a promising avenue toward generally capable reinforcement learning agents. Existing methods adapt curricula independently over either environment parameters (in single-agent settings) or co-player policies (in multi-agent settings). However, the strengths and weaknesses of co-players can manifest themselves differently depending on environmental features. It is thus crucial to consider the dependency between the environment and co-player when shaping a curriculum in multi-agent domains. In this work, we use this insight and extend Unsupervised Environment Design (UED) to multi-agent environments. We then introduce Multi-Agent Environment Design Strategist for Open-Ended Learning (MAESTRO), the first multi-agent UED approach for two-player zero-sum settings. MAESTRO efficiently produces adversarial, joint curricula over both environments and co-players and attains minimax-regret guarantees at Nash equilibrium. Our experiments show that MAESTRO outperforms a number of strong baselines on competitive two-player games, spanning discrete and continuous control settings.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 35 of its 56 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

35 of 56 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  7. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  8. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  9. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  10. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  11. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 206
  12. 2017Teacher-Student Curriculum Learningcited by 197
23 more
  1. 2020Prioritized Level Replaycited by 190
  2. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  3. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  4. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  5. 2022SMACv2: An Improved Benchmark for Cooperative Multi-Agent Reinforcement Learningcited by 118
  6. 2017Emergent Complexity via Multi-Agent Competitioncited by 117
  7. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  8. 2021Replay-Guided Adversarial Environment Designcited by 108
  9. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  10. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  11. 2017Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Taskcited by 66
  12. 2020Real World Games Look Like Spinning Topscited by 66
  13. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 58
  14. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  15. 2020Griddly: A platform for AI research in gamescited by 31
  16. 2019Self-Paced Contextual Reinforcement Learningcited by 26
  17. 2021Self-Paced Context Evaluation for Contextual Reinforcement Learningcited by 25
  18. 2021Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systemscited by 24
  19. 2021Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversitycited by 22
  20. 2021Deep Latent Competition: Learning to Race Using Visual Control Policies in Latent Spacecited by 20
  21. 2021Exploration-Exploitation in Multi-Agent Competition: Convergence with Bounded Rationalitycited by 20
  22. 2022Generalization in Cooperative Multi-Agent Systemscited by 18
  23. 2022GriddlyJS: A Web IDE for Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 12; 2025: 8; 2026: 4.

30 papers.

Sort:

By area:

  1. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 189
  2. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 79
  3. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  4. 2024Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Playcited by 21
  5. 2023Discovering General Reinforcement Learning Algorithms with Adversarial Environment Designcited by 16
  6. 2024Three Dogmas of Reinforcement Learningcited by 14
  7. 2025Cross-environment Cooperation Enables Zero-shot Multi-agent Coordinationcited by 14
  8. 2026SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learningcited by 13
  9. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Designcited by 11
  10. 2023minimax: Efficient Baselines for Autocurricula in JAXcited by 9
  11. 2025A Generalist Hanabi Agentcited by 5
  12. 2023Mix-ME: Quality-Diversity for Multi-Agent Learningcited by 4
18 more
  1. 2024Multi-Agent Diagnostics for Robustness via Illuminated Diversitycited by 4
  2. 2025Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environmentscited by 4
  3. 2023Learning Curricula in Open-Ended Worldscited by 2
  4. 2024Syllabus: Portable Curricula for Reinforcement Learning Agentscited by 2
  5. 2024Incentives to Build Houses, Trade Houses, or Trade House Building Skills in Simulated Worlds under Various Governing Systems or Institutions: Comparing Multi-agent Reinforcement Learning to Generative Agent-based Modelcited by 2
  6. 2026A Compositional Framework for Open-ended Intelligencecited by 2
  7. 2023Emergence of Collective Open-Ended Exploration from Decentralized Meta-Reinforcement Learningcited by 1
  8. 2023Curriculum Learning for Cooperation in Multi-Agent Reinforcement Learningcited by 1
  9. 2025Automatic Curriculum Design for Zero-Shot Human-AI Coordinationcited by 1
  10. 2025TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Designcited by 1
  11. 2025Shall We Play a Game? Language Models for Open-ended Wargamescited by 1
  12. 2025Scaling Multi-Agent Environment Co-Design with Diffusion Modelscited by 1
  13. 2026COvolve: Adversarial Co-Evolution of Large-Language-Model-Generated Policies and Environments via Two-Player Zero-Sum Gamecited by 1
  14. 2024FM3Q: Factorized Multi-Agent MiniMax Q-Learning for Two-Team Zero-Sum Markov Gamecited by 0
  15. 2024Games of Knightian Uncertainty as AGI testbedscited by 0
  16. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
  17. 2025Fly, Fail, Fix: Iterative Game Repair with Reinforcement Learning and Large Multimodal Modelscited by 0
  18. 2026Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMscited by 0