# CLUTR: Curriculum Learning via Unsupervised Task Representation Learning

arXiv: https://arxiv.org/abs/2210.10243 · HTML: https://mlsanity.com/paper/2210.10243/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 55 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Improving Sample Efficiency in Model-Free Reinforcement Learning from Images](https://mlsanity.com/paper/1910.01741/) (2019)
- [Learning to Drive in a Day](https://mlsanity.com/paper/1807.00412/) (2018)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Visual Reinforcement Learning with Imagined Goals](https://mlsanity.com/paper/1807.04742/) (2018)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Automatic Curriculum Learning For Deep RL: A Short Survey](https://mlsanity.com/paper/2003.04664/) (2020)
- [Replay-Guided Adversarial Environment Design](https://mlsanity.com/paper/2110.02439/) (2021)
- [Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions](https://mlsanity.com/paper/2003.08536/) (2020)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning](https://mlsanity.com/paper/1806.07937/) (2018)
- [PixelVAE: A Latent Variable Model for Natural Images](https://mlsanity.com/paper/1611.05013/) (2016)
- [Neuroevolution of Self-Interpretable Agents](https://mlsanity.com/paper/2003.08165/) (2020)
- [On the Out-of-distribution Generalization of Probabilistic Image Modelling](https://mlsanity.com/paper/2109.02639/) (2021)
- [Environment Generation for Zero-Shot Compositional Reinforcement Learning](https://mlsanity.com/paper/2201.08896/) (2022)
- [Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation](https://mlsanity.com/paper/2210.10195/) (2022)
- [Creativity: Generating Diverse Questions using Variational Autoencoders](https://mlsanity.com/paper/1704.03493/) (2017)
- [Policy-Gradient Algorithms Have No Guarantees of Convergence in Linear Quadratic Games](https://mlsanity.com/paper/1907.03712/) (2019)
- [Supervision via Competition: Robot Adversaries for Learning Tasks](https://mlsanity.com/paper/1610.01685/) (2016)

## Built on by (most cited first, 20 of 20)

- [DRED: Zero-Shot Transfer in Reinforcement Learning via Data-Regularised Environment Design](https://mlsanity.com/paper/2402.03479/) (2024)
- [Adversarial Environment Design via Regret-Guided Diffusion Models](https://mlsanity.com/paper/2410.19715/) (2024)
- [An Optimisation Framework for Unsupervised Environment Design](https://mlsanity.com/paper/2505.20659/) (2025)
- [AdaAugment: A Tuning-Free and Adaptive Approach to Enhance Data Augmentation](https://mlsanity.com/paper/2405.11467/) (2024)
- [Grounded Curriculum Learning](https://mlsanity.com/paper/2409.19816/) (2024)
- [From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning](https://mlsanity.com/paper/2606.17682/) (2026)
- [Improving Environment Novelty Quantification for Effective Unsupervised Environment Design](https://mlsanity.com/paper/2502.05726/) (2025)
- [Integrating Curricula with Replays: Its Effects on Continual Learning](https://mlsanity.com/paper/2307.05747/) (2023)
- [GACL: Grounded Adaptive Curriculum Learning with Active Task and Performance Monitoring](https://mlsanity.com/paper/2508.02988/) (2025)
- [Improving Environment Robustness of Deep Reinforcement Learning Approaches for Autonomous Racing Using Bayesian Optimization-based Curriculum Learning](https://mlsanity.com/paper/2312.10557/) (2023)
- [Reinforcement Learning via Auxiliary Task Distillation](https://mlsanity.com/paper/2406.17168/) (2024)
- [TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design](https://mlsanity.com/paper/2506.19997/) (2025)
- [Improving Regret Approximation for Unsupervised Dynamic Environment Generation](https://mlsanity.com/paper/2601.14957/) (2026)
- [Active Curriculum Refinement for Reinforcement Learning](https://mlsanity.com/paper/2608.26469/) (2026)
- [How the level sampling process impacts zero-shot generalisation in deep reinforcement learning](https://mlsanity.com/paper/2310.03494/) (2023)
- [Generative Modeling for Robust Deep Reinforcement Learning on the Traveling Salesman Problem](https://mlsanity.com/paper/2508.08718/) (2025)
- [Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learning](https://mlsanity.com/paper/2602.09813/) (2026)
- [Curriculum reinforcement learning with measurable task representation learning](https://mlsanity.com/paper/2605.23372/) (2026)
- [Manifold Bandits: Bayesian Curriculum Learning over the Latent Geometry of Large Language Models](https://mlsanity.com/paper/2606.19750/) (2026)
- [Curriculum Generation under Structured Parametric Environments for Robust Navigation Policies](https://mlsanity.com/paper/2608.08545/) (2026)

## Abstract (arXiv)

Reinforcement Learning (RL) algorithms are often known for sample inefficiency and difficult generalization. Recently, Unsupervised Environment Design (UED) emerged as a new paradigm for zero-shot generalization by simultaneously learning a task distribution and agent policies on the generated tasks. This is a non-stationary process where the task distribution evolves along with agent policies; creating an instability over time. While past works demonstrated the potential of such approaches, sampling effectively from the task space remains an open challenge, bottlenecking these approaches. To this end, we introduce CLUTR: a novel unsupervised curriculum learning algorithm that decouples task representation and curriculum learning into a two-stage optimization. It first trains a recurrent variational autoencoder on randomly generated tasks to learn a latent task manifold. Next, a teacher agent creates a curriculum by maximizing a minimax REGRET-based objective on a set of latent tasks sampled from this manifold. Using the fixed-pretrained task manifold, we show that CLUTR successfully overcomes the non-stationarity problem and improves stability. Our experimental results show CLUTR outperforms PAIRED, a principled and popular UED method, in the challenging CarRacing and navigation environments: achieving 10.6X and 45% improvement in zero-shot generalization, respectively. CLUTR also performs comparably to the non-UED state-of-the-art for CarRacing, while requiring 500X fewer environment interactions.