mlsanityData to 7 Oct 2026

No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery

arXiv 2408.15099 ↗First posted 27 Aug 2024cs.LG cs.AI cs.ROcited by 29

Not on the map. Why?

Abstract (arXiv)

What data or environments to use for training to improve downstream performance is a longstanding and very topical question in reinforcement learning. In particular, Unsupervised Environment Design (UED) methods have gained recent attention as their adaptive curricula promise to enable agents to be robust to in- and out-of-distribution tasks. This work investigates how existing UED methods select training environments, focusing on task prioritisation metrics. Surprisingly, despite methods aiming to maximise regret in theory, the practical approximations do not correlate with regret but with success rate. As a result, a significant portion of an agent's experience comes from environments it has already mastered, offering little to no contribution toward enhancing its abilities. Put differently, current methods fail to predict intuitive measures of ``learnability.'' Specifically, they are unable to consistently identify those scenarios that the agent can sometimes solve, but not always. Based on our analysis, we develop a method that directly trains on scenarios with high learnability. This simple and intuitive approach outperforms existing UED methods in several binary-outcome environments, including the standard domain of Minigrid and a novel setting closely inspired by a real-world robotics problem. We further introduce a new adversarial evaluation procedure for directly measuring robustness, closely mirroring the conditional value at risk (CVaR). We open-source all our code and present visualisations of final policies here: https://github.com/amacrutherford/sampling-for-learnability.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 43 of its 64 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

43 of 64 references.

References · 43

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2020Mastering Atari with Discrete World Modelscited by 990
  7. 2017Robust Adversarial Reinforcement Learningcited by 439
  8. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  9. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  10. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  11. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  12. 2019Multi-Agent Pathfinding: Definitions, Variants, and Benchmarkscited by 197
31 more
  1. 2020Prioritized Level Replaycited by 190
  2. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  3. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  4. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  5. 2019Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environmentscited by 113
  6. 2021Replay-Guided Adversarial Environment Designcited by 108
  7. 2022Discovered Policy Optimisationcited by 103
  8. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  9. 2017Towards Optimally Decentralized Multi-Robot Collision Avoidance via Deep Reinforcement Learningcited by 81
  10. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  11. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 54
  12. 2020Motion Planning and Control for Mobile Robot Navigation Using Machine Learning: a Surveycited by 48
  13. 2019Collision Avoidance in Pedestrian-Rich Environments with Deep Reinforcement Learningcited by 44
  14. 2023MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learningcited by 30
  15. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  16. 2023DRL-VO: Learning to Navigate Through Crowded Dynamic Scenes Using Velocity Obstaclescited by 25
  17. 2023Proximal Curriculum for Reinforcement Learning Agentscited by 19
  18. 2023On the Foundation of Distributionally Robust Reinforcement Learningcited by 18
  19. 2023Stabilizing Unsupervised Environment Design with a Learned Adversarycited by 17
  20. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  21. 2019Distributionally Robust Partially Observable Markov Decision Process with Moment-based Ambiguitycited by 15
  22. 2022Benchmarking Reinforcement Learning Techniques for Autonomous Navigationcited by 15
  23. 2023Reward-Free Curricula for Training Robust World Modelscited by 13
  24. 2020Distributionally Robust Markov Decision Processes and their Connection to Risk Measurescited by 12
  25. 2022Reinforcement Learned Distributed Multi-Robot Navigation with Reciprocal Velocity Obstacle Shaped Rewardscited by 11
  26. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
  27. 2024JaxUED: A simple and useable UED library in Jaxcited by 10
  28. 2023minimax: Efficient Baselines for Autocurricula in JAXcited by 9
  29. 2024Proximal Curriculum with Task Correlations for Deep Reinforcement Learningcited by 6
  30. 2019DeepMNavigate: Deep Reinforced Multi-Robot Navigation Unifying Local & Global Collision Avoidancecited by 5
  31. 2022Learning Perceptual Hallucination for Multi-Robot Navigation in Narrow Hallwayscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 15; 2026: 12.

29 papers.

Sort:

By area:

  1. 2024Kinetix: Investigating the Training of General Agents through Open-Ended Physics-Based Control Taskscited by 40
  2. 2025Learning to Reason at the Frontier of Learnabilitycited by 20
  3. 2025Bootstrapping Task Spaces for Self-Improvementcited by 10
  4. 2025An Optimisation Framework for Unsupervised Environment Designcited by 9
  5. 2025MEAL: A Benchmark for Continual Multi-Agent Reinforcement Learningcited by 8
  6. 2025ROTATE: Regret-driven Open-ended Training for Ad Hoc Teamworkcited by 6
  7. 2025BuilderBench: The Building Blocks of Intelligent Agentscited by 4
  8. 2026GASP: Guided Asymmetric Self-Play For Coding LLMscited by 4
  9. 2025Improving Human-AI Coordination through Online Adversarial Training and Generative Modelscited by 3
  10. 2025Efficient Skill Discovery via Regret-Aware Optimizationcited by 3
  11. 2026Preventing Learning Stagnation in PPO by Scaling to 1 Million Parallel Environmentscited by 3
  12. 2024Syllabus: Portable Curricula for Reinforcement Learning Agentscited by 2
17 more
  1. 2025Unsupervised Partner Design Enables Robust Ad-hoc Teamworkcited by 2
  2. 2025CAMAR: Continuous Actions Multi-Agent Routingcited by 2
  3. 2025Beyond Fixed Tasks: Unsupervised Environment Design for Task-Level Pairscited by 2
  4. 2025Automatic Curriculum Design for Zero-Shot Human-AI Coordinationcited by 1
  5. 2025TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Designcited by 1
  6. 2025Mitigating Goal Misgeneralization via Minimax Regretcited by 1
  7. 2026Improving Regret Approximation for Unsupervised Dynamic Environment Generationcited by 1
  8. 2026IR-SIM: A Lightweight Declarative Simulator for Navigation Learning and Benchmarkingcited by 1
  9. 2026Scaling Curriculum Learning For Autonomous Drivingcited by 1
  10. 2026Active Curriculum Refinement for Reinforcement Learningcited by 1
  11. 2025Towards a Formal Theory of the Need for Competence via Computational Intrinsic Motivationcited by 0
  12. 2026D'ej`aQ: Open-Ended Evolution of Diverse, Learnable and Verifiable Problemscited by 0
  13. 2026TABX: A High-Throughput Sandbox Battle Simulator for Multi-Agent Reinforcement Learningcited by 0
  14. 2026Solving Parameter-Robust Avoid Problems with Unknown Feasibility using Reinforcement Learningcited by 0
  15. 2026Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learningcited by 0
  16. 2026Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisationcited by 0
  17. 2026Open-ended Multi-agent Autocurricula via Visual Inspection of Policies with Multi-modal LLMscited by 0