mlsanityData to 14 Oct 2026

Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairs

arXiv 2511.12706 ↗First posted 16 Nov 2025cs.LG cs.AIcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Training general agents to follow complex instructions (tasks) in intricate environments (levels) remains a core challenge in reinforcement learning. Random sampling of task-level pairs often produces unsolvable combinations, highlighting the need to co-design tasks and levels. While unsupervised environment design (UED) has proven effective at automatically designing level curricula, prior work has only considered a fixed task. We present ATLAS (Aligning Tasks and Levels for Autocurricula of Specifications), a novel method that generates joint autocurricula over tasks and levels. Our approach builds upon UED to automatically produce solvable yet challenging task-level pairs for policy training. To evaluate ATLAS and drive progress in the field, we introduce an evaluation suite that models tasks as reward machines in Minigrid levels. Experiments demonstrate that ATLAS vastly outperforms random sampling approaches, particularly when sampling solvable pairs is unlikely. We further show that mutations leveraging the structure of both tasks and levels accelerate convergence to performant policies.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 49 of its 78 references that are in the map. First posted on arXiv 16 Nov 2025.

Builds on

49 of 78 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025An Optimisation Framework for Unsupervised Environment Design27 May
  2. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery27 Aug
  3. 2024Refining Minimax Regret for Unsupervised Environment Design19 Feb
  4. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design5 Feb

Other references · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  5. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  6. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  7. 2017Hindsight Experience Replaycited by 1,136
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  9. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  10. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 373
  11. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 327
  12. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 275
33 more
  1. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
  2. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  3. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 233
  4. 2017Teacher-Student Curriculum Learningcited by 210
  5. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 203
  6. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 192
  7. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  8. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  9. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  10. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 150
  11. 2024BALROG: Benchmarking Agentic LLM and VLM Reasoning On Gamescited by 146
  12. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 141
  13. 2021Replay-Guided Adversarial Environment Designcited by 111
  14. 2019Graph Normalizing Flowscited by 104
  15. 2020PCGRL: Procedural Content Generation via Reinforcement Learningcited by 90
  16. 2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 85
  17. 2021LTL2Action: Generalizing LTL Instructions for Multi-Task RLcited by 74
  18. 2024OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Codecited by 74
  19. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 72
  20. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 61
  21. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  22. 2023Grounding Complex Natural Language Commands for Temporal Tasks in Unseen Environmentscited by 53
  23. 2022Improving Intrinsic Exploration with Language Abstractionscited by 49
  24. 2020Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL formulascited by 36
  25. 2020Reward Machines for Cooperative Multi-Agent Reinforcement Learningcited by 35
  26. 2020Induction and Exploitation of Subgoal Automata for Reinforcement Learningcited by 26
  27. 2024MaestroMotif: Skill Design from Artificial Intelligence Feedbackcited by 24
  28. 2019Learning Compositional Neural Programs with Recursive Tree Search and Planningcited by 17
  29. 2022Learning to Follow Instructions in Text-Based Gamescited by 12
  30. 2023Generalization through Diversity: Improving Unsupervised Environment Designcited by 9
  31. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Datacited by 5
  32. 2025Improving Environment Novelty Quantification for Effective Unsupervised Environment Designcited by 4
  33. 2025Imagined Autocurriculacited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026FrogNano: Training a 4B Coding Agent via Online Task Synthesiscited by 4
  2. 2026Scalable Environments Drive Generalizable Agentscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design
  2. 2026Improving Regret Approximation for Unsupervised Dynamic Environment Generation
  3. 2026Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learning
  4. 2025An Empirical Study of Deep Reinforcement Learning in Continuing Tasks
  5. 2025Towards Principled Unsupervised Multi-Agent Reinforcement Learning
  6. 2026Unsupervised Learning of Efficient Exploration: Pre-training Adaptive Policies via Self-Imposed Goals
  7. 2024The Overcooked Generalisation Challenge: Evaluating Cooperation with Novel Partners in Unknown Environments Using Unsupervised Environment Design
  8. 2026FARL: A Fully Automated Pipeline of RL Interface and Solver Design