# Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments

arXiv: https://arxiv.org/abs/1910.07224 · HTML: https://mlsanity.com/paper/1910.07224/

## Key facts

- Cited by 119 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2019.

## Built on by (most cited first, 30 of 119)

- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Efficient Reinforcement Finetuning via Adaptive Curriculum Learning](https://mlsanity.com/paper/2504.05520/) (2025)
- [OMNI: Open-endedness via Models of human Notions of Interestingness](https://mlsanity.com/paper/2306.01711/) (2023)
- [Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay](https://mlsanity.com/paper/2506.05316/) (2025)
- [Exploration via Elliptical Episodic Bonuses](https://mlsanity.com/paper/2210.05805/) (2022)
- [Improving Intrinsic Exploration with Language Abstractions](https://mlsanity.com/paper/2202.08938/) (2022)
- [Let's Learn Step by Step: Enhancing In-Context Learning Ability with Curriculum Learning](https://mlsanity.com/paper/2402.10738/) (2024)
- [DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training](https://mlsanity.com/paper/2504.09710/) (2025)
- [Learning Controllable Content Generators](https://mlsanity.com/paper/2105.02993/) (2021)
- [Adversarial Reinforcement Learning for Procedural Content Generation](https://mlsanity.com/paper/2103.04847/) (2021)
- [Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability](https://mlsanity.com/paper/2209.08025/) (2022)
- [Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft](https://mlsanity.com/paper/2106.14876/) (2021)
- [Environment Generation for Zero-Shot Compositional Reinforcement Learning](https://mlsanity.com/paper/2201.08896/) (2022)
- [MAESTRO: Open-Ended Environment Design for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2303.03376/) (2023)
- [No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery](https://mlsanity.com/paper/2408.15099/) (2024)
- [Training a Generally Curious Agent](https://mlsanity.com/paper/2502.17543/) (2025)
- [Curriculum Reinforcement Learning using Optimal Transport via Gradual Domain Adaptation](https://mlsanity.com/paper/2210.10195/) (2022)
- [Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems](https://mlsanity.com/paper/2111.04613/) (2021)
- [A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning](https://mlsanity.com/paper/2102.13176/) (2021)
- [Solving Compositional Reinforcement Learning Problems via Task Reduction](https://mlsanity.com/paper/2103.07607/) (2021)
- [Hypernetworks for Zero-shot Transfer in Reinforcement Learning](https://mlsanity.com/paper/2211.15457/) (2022)
- [TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RL](https://mlsanity.com/paper/2103.09815/) (2021)
- [Refining Minimax Regret for Unsupervised Environment Design](https://mlsanity.com/paper/2402.12284/) (2024)
- [FLD: Fourier Latent Dynamics for Structured Motion Representation and Learning](https://mlsanity.com/paper/2402.13820/) (2024)
- [Understanding the Complexity Gains of Single-Task RL with a Curriculum](https://mlsanity.com/paper/2212.12809/) (2022)
- [Grounding Aleatoric Uncertainty for Unsupervised Environment Design](https://mlsanity.com/paper/2207.05219/) (2022)
- [Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation](https://mlsanity.com/paper/2301.11741/) (2023)

## Abstract (arXiv)

We consider the problem of how a teacher algorithm can enable an unknown Deep Reinforcement Learning (DRL) student to become good at a skill over a wide range of diverse environments. To do so, we study how a teacher algorithm can learn to generate a learning curriculum, whereby it sequentially samples parameters controlling a stochastic procedural generation of environments. Because it does not initially know the capacities of its student, a key challenge for the teacher is to discover which environments are easy, difficult or unlearnable, and in what order to propose them to maximize the efficiency of learning over the learnable ones. To achieve this, this problem is transformed into a surrogate continuous bandit problem where the teacher samples environments in order to maximize absolute learning progress of its student. We present a new algorithm modeling absolute learning progress with Gaussian mixture models (ALP-GMM). We also adapt existing algorithms and provide a complete study in the context of DRL. Using parameterized variants of the BipedalWalker environment, we study their efficiency to personalize a learning curriculum for different learners (embodiments), their robustness to the ratio of learnable/unlearnable environments, and their scalability to non-linear and high-dimensional parameter spaces. Videos and code are available at https://github.com/flowersteam/teachDeepRL.