mlsanityData to 7 Oct 2026

AutoDIME: Automatic Design of Interesting Multi-Agent Environments

arXiv 2203.02481 ↗First posted 4 Mar 2022cs.LG cs.AI cs.MAcited by 0

Not on the map. Why?

Abstract (arXiv)

Designing a distribution of environments in which RL agents can learn interesting and useful skills is a challenging and poorly understood task, for multi-agent environments the difficulties are only exacerbated. One approach is to train a second RL agent, called a teacher, who samples environments that are conducive for the learning of student agents. However, most previous proposals for teacher rewards do not generalize straightforwardly to the multi-agent setting. We examine a set of intrinsic teacher rewards derived from prediction problems that can be applied in multi-agent settings and evaluate them in Mujoco tasks such as multi-agent Hide and Seek as well as a diagnostic single-agent maze task. Of the intrinsic rewards considered we found value disagreement to be most consistent across tasks, leading to faster and more reliable emergence of advanced skills in Hide and Seek and the maze task. Another candidate intrinsic reward considered, value prediction error, also worked well in Hide and Seek but was susceptible to noisy-TV style distractions in stochastic environments. Policy disagreement performed well in the maze task but did not speed up learning in Hide and Seek. Our results suggest that intrinsic teacher rewards, and in particular value disagreement, are a promising approach for automating both single and multi-agent environment design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 58 references that are in the map. First posted on arXiv 4 Mar 2022.

Builds on

34 of 58 references.

References · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2019Solving Rubik's Cube with a Robot Handcited by 699
  10. 2016Learning to reinforcement learncited by 369
  11. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
  12. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
22 more
  1. 2019Self-Supervised Exploration via Disagreementcited by 263
  2. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  3. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  4. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  5. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  6. 2017Teacher-Student Curriculum Learningcited by 197
  7. 2020Prioritized Level Replaycited by 190
  8. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  9. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  10. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  11. 2017Emergent Complexity via Multi-Agent Competitioncited by 117
  12. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  13. 2018Model-Based Active Explorationcited by 105
  14. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  15. 2020Automatic Curriculum Learning through Value Disagreementcited by 86
  16. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  17. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  18. 2018DORA The Explorer: Directed Outreaching Reinforcement Action-Selectioncited by 27
  19. 2018Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learningcited by 23
  20. 2019Competitive Experience Replaycited by 23
  21. 2017Sim2Real View Invariant Visual Servoing by Recurrent Controlcited by 19
  22. 2021Adversarial Environment Generation for Learning to Navigate the Webcited by 13

Built on by

Not cited yet.