# Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft

arXiv: https://arxiv.org/abs/2106.14876 · HTML: https://mlsanity.com/paper/2106.14876/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 28 Jun 2021.

## Builds on (most cited first, 22 of 22)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Unifying Count-Based Exploration and Intrinsic Motivation](https://mlsanity.com/paper/1606.01868/) (2016)
- [Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design](https://mlsanity.com/paper/2012.02096/) (2020)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments](https://mlsanity.com/paper/1910.07224/) (2019)
- [Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions](https://mlsanity.com/paper/2003.08536/) (2020)
- [Learning with AMIGo: Adversarially Motivated Intrinsic Goals](https://mlsanity.com/paper/2006.12122/) (2020)
- [Automatic Curriculum Learning through Value Disagreement](https://mlsanity.com/paper/2006.09641/) (2020)
- [Asymmetric self-play for automatic goal discovery in robotic manipulation](https://mlsanity.com/paper/2101.04882/) (2021)
- [Active World Model Learning with Progress Curiosity](https://mlsanity.com/paper/2007.07853/) (2020)
- [Learning Goal Embeddings via Self-Play for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1811.09083/) (2018)
- [Adversarial Environment Generation for Learning to Navigate the Web](https://mlsanity.com/paper/2103.01991/) (2021)
- [Deep Curiosity Search: Intra-Life Exploration Can Improve Performance on Challenging Deep Reinforcement Learning Problems](https://mlsanity.com/paper/1806.00553/) (2018)

## Built on by (most cited first, 30 of 35)

- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [Darwin Godel Machine: Open-Ended Evolution of Self-Improving Agents](https://mlsanity.com/paper/2505.22954/) (2025)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [OMNI: Open-endedness via Models of human Notions of Interestingness](https://mlsanity.com/paper/2306.01711/) (2023)
- [Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft](https://mlsanity.com/paper/2312.09238/) (2023)
- [Skill Reinforcement Learning and Planning for Open-World Long-Horizon Tasks](https://mlsanity.com/paper/2303.16563/) (2023)
- [S-Agents: Self-organizing Agents in Open-ended Environments](https://mlsanity.com/paper/2402.04578/) (2024)
- [Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds](https://mlsanity.com/paper/2210.13417/) (2022)
- [Eurekaverse: Environment Curriculum Generation via Large Language Models](https://mlsanity.com/paper/2411.01775/) (2024)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [Cross-Episodic Curriculum for Transformer Agents](https://mlsanity.com/paper/2310.08549/) (2023)
- [Hieros: Hierarchical Imagination on Structured State Space Sequence World Models](https://mlsanity.com/paper/2310.05167/) (2023)
- [Masked Generative Priors Improve World Models Sequence Modelling Capabilities](https://mlsanity.com/paper/2410.07836/) (2024)
- [MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces](https://mlsanity.com/paper/2502.07709/) (2025)
- [Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems](https://mlsanity.com/paper/2310.10462/) (2023)
- [Interleaved Multitask Learning with Energy Modulated Learning Progress](https://mlsanity.com/paper/2504.00707/) (2025)
- [Interestingness as an Inductive Heuristic for Future Compression Progress](https://mlsanity.com/paper/2605.14831/) (2026)
- [Teacher-student curriculum learning for reinforcement learning](https://mlsanity.com/paper/2210.17368/) (2022)
- [Learning Curricula in Open-Ended Worlds](https://mlsanity.com/paper/2312.03126/) (2023)
- [Syllabus: Portable Curricula for Reinforcement Learning Agents](https://mlsanity.com/paper/2411.11318/) (2024)
- [DyMoDreamer: World Modeling with Dynamic Modulation](https://mlsanity.com/paper/2509.24804/) (2025)
- [Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies](https://mlsanity.com/paper/2509.05735/) (2025)
- [Scaling Curriculum Learning For Autonomous Driving](https://mlsanity.com/paper/2608.22549/) (2026)
- [Massively Multiagent Minigames for Training Generalist Agents](https://mlsanity.com/paper/2406.05071/) (2024)

## Abstract (arXiv)

An important challenge in reinforcement learning is training agents that can solve a wide variety of tasks. If tasks depend on each other (e.g. needing to learn to walk before learning to run), curriculum learning can speed up learning by focusing on the next best task to learn. We explore curriculum learning in a complex, visual domain with many hard exploration challenges: Minecraft. We find that learning progress (defined as a change in success probability of a task) is a reliable measure of learnability for automatically constructing an effective curriculum. We introduce a learning-progress based curriculum and test it on a complex reinforcement learning problem (called "Simon Says") where an agent is instructed to obtain a desired goal item. Many of the required skills depend on each other. Experiments demonstrate that: (1) a within-episode exploration bonus for obtaining new items improves performance, (2) dynamically adjusting this bonus across training such that it only applies to items the agent cannot reliably obtain yet further increases performance, (3) the learning-progress based curriculum elegantly follows the learning curve of the agent, and (4) when the learning-progress based curriculum is combined with the dynamic exploration bonus it learns much more efficiently and obtains far higher performance than uniform baselines. These results suggest that combining intra-episode and across-training exploration bonuses with learning progress creates a promising method for automated curriculum generation, which may substantially increase our ability to train more capable, generally intelligent agents.