# TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL

arXiv: https://arxiv.org/abs/2103.09815 · HTML: https://mlsanity.com/paper/2103.09815/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 80 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 30 of 48)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Wasserstein GAN](https://mlsanity.com/paper/1701.07875/) (2017)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Meta-Learning: A Survey](https://mlsanity.com/paper/1810.03548/) (2018)
- [BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learning](https://mlsanity.com/paper/1810.08272/) (2018)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [A Study on Overfitting in Deep Reinforcement Learning](https://mlsanity.com/paper/1804.06893/) (2018)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [Automatic Curriculum Learning For Deep RL: A Short Survey](https://mlsanity.com/paper/2003.04664/) (2020)
- [Multi-task Deep Reinforcement Learning with PopArt](https://mlsanity.com/paper/1809.04474/) (2018)
- [EPOpt: Learning Robust Neural Network Policies Using Model Ensembles](https://mlsanity.com/paper/1610.01283/) (2016)
- [Behaviour Suite for Reinforcement Learning](https://mlsanity.com/paper/1908.03568/) (2019)

## Built on by (most cited first, 18 of 18)

- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation](https://mlsanity.com/paper/2210.10195/) (2022)
- [CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning](https://mlsanity.com/paper/2110.02102/) (2021)
- [Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL](https://mlsanity.com/paper/2106.15883/) (2021)
- [Proximal Curriculum with Task Correlations for Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02481/) (2024)
- [MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces](https://mlsanity.com/paper/2502.07709/) (2025)
- [On the Benefit of Optimal Transport for Curriculum Reinforcement Learning](https://mlsanity.com/paper/2309.14091/) (2023)
- [GriddlyJS: A Web IDE for Reinforcement Learning](https://mlsanity.com/paper/2207.06105/) (2022)
- [Improving Generalization in Game Agents with Data Augmentation in Imitation Learning](https://mlsanity.com/paper/2309.12815/) (2023)
- [Continual Driving Policy Optimization with Closed-Loop Individualized Curricula](https://mlsanity.com/paper/2309.14209/) (2023)
- [Automatic Curriculum Learning with Gradient Reward Signals](https://mlsanity.com/paper/2312.13565/) (2023)
- [Heterogeneous Adversarial Play in Interactive Environments](https://mlsanity.com/paper/2510.18407/) (2025)
- [Sample Efficient Myopic Exploration Through Multitask Reinforcement Learning with Diverse Tasks](https://mlsanity.com/paper/2403.01636/) (2024)
- [Syllabus: Portable Curricula for Reinforcement Learning Agents](https://mlsanity.com/paper/2411.11318/) (2024)
- [C-Procgen: Empowering Procgen with Controllable Contexts](https://mlsanity.com/paper/2311.07312/) (2023)
- [Rethinking Teacher-Student Curriculum Learning through the Cooperative Mechanics of Experience](https://mlsanity.com/paper/2404.03084/) (2024)
- [TRACED: Transition-aware Regret Approximation with Co-learnability for Environment Design](https://mlsanity.com/paper/2506.19997/) (2025)
- [Causal-Paced Deep Reinforcement Learning](https://mlsanity.com/paper/2507.02910/) (2025)

## Abstract (arXiv)

Training autonomous agents able to generalize to multiple tasks is a key target of Deep Reinforcement Learning (DRL) research. In parallel to improving DRL algorithms themselves, Automatic Curriculum Learning (ACL) study how teacher algorithms can train DRL agents more efficiently by adapting task selection to their evolving abilities. While multiple standard benchmarks exist to compare DRL agents, there is currently no such thing for ACL algorithms. Thus, comparing existing approaches is difficult, as too many experimental parameters differ from paper to paper. In this work, we identify several key challenges faced by ACL algorithms. Based on these, we present TeachMyAgent (TA), a benchmark of current ACL algorithms leveraging procedural task generation. It includes 1) challenge-specific unit-tests using variants of a procedural Box2D bipedal walker environment, and 2) a new procedural Parkour environment combining most ACL challenges, making it ideal for global performance assessment. We then use TeachMyAgent to conduct a comparative study of representative existing approaches, showcasing the competitiveness of some ACL algorithms that do not use expert knowledge. We also show that the Parkour environment remains an open problem. We open-source our environments, all studied ACL algorithms (collected from open-source code or re-implemented), and DRL students in a Python package available at https://github.com/flowersteam/TeachMyAgent.