# Transferable Curricula through Difficulty Conditioned Generators

arXiv: https://arxiv.org/abs/2306.13028 · HTML: https://mlsanity.com/paper/2306.13028/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 22 Jun 2023.

## Builds on (most cited first, 16 of 16)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Evolving Curricula with Regret-Based Environment Design](https://mlsanity.com/paper/2203.01302/) (2022)
- [Replay-Guided Adversarial Environment Design](https://mlsanity.com/paper/2110.02439/) (2021)
- [AlphaStar: An Evolutionary Computation Perspective](https://mlsanity.com/paper/1902.01724/) (2019)
- [Variational Item Response Theory: Fast, Accurate, and Expressive](https://mlsanity.com/paper/2002.00276/) (2020)
- [Automated curricula through setter-solver interactions](https://mlsanity.com/paper/1909.12892/) (2019)
- [It Takes Four to Tango: Multiagent Selfplay for Automatic Curriculum Generation](https://mlsanity.com/paper/2202.10608/) (2022)
- [Generalization through Diversity: Improving Unsupervised Environment Design](https://mlsanity.com/paper/2301.08025/) (2023)
- [An Optimization Framework for Task Sequencing in Curriculum Learning](https://mlsanity.com/paper/1901.11478/) (2019)

## Built on by (most cited first, 4 of 4)

- [Enhancing the Hierarchical Environment Design via Generative Trajectory Modeling](https://mlsanity.com/paper/2310.00301/) (2023)
- [Training Reinforcement Learning Agents and Humans With Difficulty-Conditioned Generators](https://mlsanity.com/paper/2312.02309/) (2023)
- [Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learning](https://mlsanity.com/paper/2602.09813/) (2026)
- [Generative Flow on Symbolic Abstractions for Embodiment Capability Discovery](https://mlsanity.com/paper/2610.17174/) (2026)

## Abstract (arXiv)

Advancements in reinforcement learning (RL) have demonstrated superhuman performance in complex tasks such as Starcraft, Go, Chess etc. However, knowledge transfer from Artificial "Experts" to humans remain a significant challenge. A promising avenue for such transfer would be the use of curricula. Recent methods in curricula generation focuses on training RL agents efficiently, yet such methods rely on surrogate measures to track student progress, and are not suited for training robots in the real world (or more ambitiously humans). In this paper, we introduce a method named Parameterized Environment Response Model (PERM) that shows promising results in training RL agents in parameterized environments. Inspired by Item Response Theory, PERM seeks to model difficulty of environments and ability of RL agents directly. Given that RL agents and humans are trained more efficiently under the "zone of proximal development", our method generates a curriculum by matching the difficulty of an environment to the current ability of the student. In addition, PERM can be trained offline and does not employ non-stationary measures of student ability, making it suitable for transfer between students. We demonstrate PERM's ability to represent the environment parameter space, and training with RL agents with PERM produces a strong performance in deterministic environments. Lastly, we show that our method is transferable between students, without any sacrifice in training quality.