mlsanityData to 7 Oct 2026

CLUTR: Curriculum Learning via Unsupervised Task Representation Learning

arXiv 2210.10243 ↗First posted 19 Oct 2022cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning (RL) algorithms are often known for sample inefficiency and difficult generalization. Recently, Unsupervised Environment Design (UED) emerged as a new paradigm for zero-shot generalization by simultaneously learning a task distribution and agent policies on the generated tasks. This is a non-stationary process where the task distribution evolves along with agent policies; creating an instability over time. While past works demonstrated the potential of such approaches, sampling effectively from the task space remains an open challenge, bottlenecking these approaches. To this end, we introduce CLUTR: a novel unsupervised curriculum learning algorithm that decouples task representation and curriculum learning into a two-stage optimization. It first trains a recurrent variational autoencoder on randomly generated tasks to learn a latent task manifold. Next, a teacher agent creates a curriculum by maximizing a minimax REGRET-based objective on a set of latent tasks sampled from this manifold. Using the fixed-pretrained task manifold, we show that CLUTR successfully overcomes the non-stationarity problem and improves stability. Our experimental results show CLUTR outperforms PAIRED, a principled and popular UED method, in the challenging CarRacing and navigation environments: achieving 10.6X and 45% improvement in zero-shot generalization, respectively. CLUTR also performs comparably to the non-UED state-of-the-art for CarRacing, while requiring 500X fewer environment interactions.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 19 Oct 2022.

Builds on

33 of 55 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016OpenAI Gymcited by 1,684
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2015Generating Sentences from a Continuous Spacecited by 607
  7. 2020Curriculum Learning for Reinforcement Learning Domains: A Framework and Surveycited by 350
  8. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  9. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  10. 2018Learning to Drive in a Daycited by 266
  11. 2020Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Designcited by 260
  12. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
21 more
  1. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  2. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  3. 2020Prioritized Level Replaycited by 190
  4. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  5. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  6. 2020Automatic Curriculum Learning For Deep RL: A Short Surveycited by 141
  7. 2021Replay-Guided Adversarial Environment Designcited by 108
  8. 2020Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutionscited by 103
  9. 2019Unsupervised speech representation learning using WaveNet autoencoderscited by 90
  10. 2018A Dissection of Overfitting and Generalization in Continuous Reinforcement Learningcited by 86
  11. 2016PixelVAE: A Latent Variable Model for Natural Imagescited by 69
  12. 2020Neuroevolution of Self-Interpretable Agentscited by 48
  13. 2021On the Out-of-distribution Generalization of Probabilistic Image Modellingcited by 32
  14. 2022Environment Generation for Zero-Shot Compositional Reinforcement Learningcited by 32
  15. 2022Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptationcited by 28
  16. 2017Creativity: Generating Diverse Questions using Variational Autoencoderscited by 24
  17. 2016Supervision via Competition: Robot Adversaries for Learning Taskscited by 15
  18. 2019Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear Quadratic Gamescited by 15
  19. 2020Robust Output Regulation of the Linearized Boussinesq Equations with Boundary Control and Observationcited by 6
  20. 2021Scenic4RL: Programmatic Modeling and Generation of Reinforcement Learning Environmentscited by 6
  21. 2022Improving VAE-based Representation Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 5; 2026: 6.

19 papers.

Sort:

By area:

  1. 2024DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Designcited by 11
  2. 2024Adversarial Environment Design via Regret-Guided Diffusion Modelscited by 9
  3. 2025An Optimisation Framework for Unsupervised Environment Designcited by 9
  4. 2024AdaAugment: A Tuning-Free and Adaptive Approach to Enhance Data Augmentationcited by 7
  5. 2024Grounded Curriculum Learningcited by 5
  6. 2026From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoningcited by 5
  7. 2025Improving Environment Novelty Quantification for Effective Unsupervised Environment Designcited by 4
  8. 2023Integrating Curricula with Replays: Its Effects on Continual Learningcited by 3
  9. 2025GACL: Grounded Adaptive Curriculum Learning with Active Task and Performance Monitoringcited by 3
  10. 2023Improving Environment Robustness of Deep Reinforcement Learning Approaches for Autonomous Racing Using Bayesian Optimization-based Curriculum Learningcited by 2
  11. 2024Reinforcement Learning via Auxiliary Task Distillationcited by 2
  12. 2025TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Designcited by 1
7 more
  1. 2026Improving Regret Approximation for Unsupervised Dynamic Environment Generationcited by 1
  2. 2026Active Curriculum Refinement for Reinforcement Learningcited by 1
  3. 2023How the level sampling process impacts zero-shot generalisation in deep reinforcement learningcited by 0
  4. 2025Generative Modeling for Robust Deep Reinforcement Learning on the Traveling Salesman Problemcited by 0
  5. 2026Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learningcited by 0
  6. 2026Curriculum reinforcement learning with measurable task representation learningcited by 0
  7. 2026Curriculum Generation under Structured Parametric Environments for Robust Navigation Policiescited by 0