mlsanityData to 13 Oct 2026

Improving Environment Novelty Quantification for Effective Unsupervised Environment Design

arXiv 2502.05726 ↗First posted 8 Feb 2025cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Unsupervised Environment Design (UED) formalizes the problem of autocurricula through interactive training between a teacher agent and a student agent. The teacher generates new training environments with high learning potential, curating an adaptive curriculum that strengthens the student's ability to handle unseen scenarios. Existing UED methods mainly rely on regret, a metric that measures the difference between the agent's optimal and actual performance, to guide curriculum design. Regret-driven methods generate curricula that progressively increase environment complexity for the student but overlook environment novelty -- a critical element for enhancing an agent's generalizability. Measuring environment novelty is especially challenging due to the underspecified nature of environment parameters in UED, and existing approaches face significant limitations. To address this, this paper introduces the Coverage-based Evaluation of Novelty In Environment (CENIE) framework. CENIE proposes a scalable, domain-agnostic, and curriculum-aware approach to quantifying environment novelty by leveraging the student's state-action space coverage from previous curriculum experiences. We then propose an implementation of CENIE that models this coverage and measures environment novelty using Gaussian Mixture Models. By integrating both regret and novelty as complementary objectives for curriculum design, CENIE facilitates effective exploration across the state-action space while progressively increasing curriculum complexity. Empirical evaluations demonstrate that augmenting existing regret-based UED algorithms with CENIE achieves state-of-the-art performance across multiple benchmarks, underscoring the effectiveness of novelty-driven autocurricula for robust generalization.

Key facts. Cited by 4 papers in the map (data through 13 Oct 2026). Builds on 35 of its 59 references that are in the map. First posted on arXiv 8 Feb 2025.

Builds on

35 of 59 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2015Variational Inference with Normalizing Flowscited by 2,163
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,940
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,411
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,180
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,093
  7. 2015Prioritized Experience Replaycited by 1,078
  8. 2018Exploration by Random Network Distillationcited by 971
  9. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 790
  10. 2021Domain Generalization: A Surveycited by 760
  11. 2019Solving Rubik's Cube with a Robot Handcited by 702
  12. 2018Learning Dexterous In-Hand Manipulationcited by 645
23 more
  1. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  2. 2017Robust Adversarial Reinforcement Learningcited by 440
  3. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  4. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 322
  5. 2018Large-Scale Study of Curiosity-Driven Learningcited by 279
  6. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 263
  7. 2017Count-Based Exploration with Neural Density Modelscited by 255
  8. 2020Prioritized Level Replaycited by 192
  9. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 186
  10. 2022Evolving Curricula with Regret-Based Environment Designcited by 168
  11. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 143
  12. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 115
  13. 2021Replay-Guided Adversarial Environment Designcited by 108
  14. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 104
  15. 2022The Role of Coverage in Online Reinforcement Learningcited by 83
  16. 2019Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Researchcited by 60
  17. 2019Simplified Action Decoder for Deep Multi-Agent Reinforcement Learningcited by 48
  18. 2019Maximum Entropy-Regularized Multi-Goal Reinforcement Learningcited by 39
  19. 2017Deep Gaussian Mixture Modelscited by 30
  20. 2019Curiosity-Driven Experience Prioritization via Density Estimationcited by 27
  21. 2022CLUTR: Curriculum Learning via Unsupervised Task Representation Learningcited by 20
  22. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  23. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025On Generalization Across Environments In Multi-Objective Reinforcement Learningcited by 8
  2. 2025Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairscited by 2
  3. 2026Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learningcited by 2
  4. 2025TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Designcited by 1