# Teacher-student curriculum learning for reinforcement learning

arXiv: https://arxiv.org/abs/2210.17368 · HTML: https://mlsanity.com/paper/2210.17368/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 24 of its 63 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [Google Research Football: A Novel Reinforcement Learning Environment](https://mlsanity.com/paper/1907.11180/) (2019)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Prioritized Level Replay](https://mlsanity.com/paper/2010.03934/) (2020)
- [Learning by Playing - Solving Sparse Reward Tasks from Scratch](https://mlsanity.com/paper/1802.10567/) (2018)
- [Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments](https://mlsanity.com/paper/1910.07224/) (2019)
- [Revisiting Design Choices in Proximal Policy Optimization](https://mlsanity.com/paper/2009.10897/) (2020)
- [Learning Curriculum Policies for Reinforcement Learning](https://mlsanity.com/paper/1812.00285/) (2018)
- [Multi-task curriculum learning in a complex, visual, hard-exploration domain: Minecraft](https://mlsanity.com/paper/2106.14876/) (2021)
- [Strategic Attentive Writer for Learning Macro-Actions](https://mlsanity.com/paper/1606.04695/) (2016)
- [StarCraft Micromanagement with Reinforcement Learning and Curriculum Transfer Learning](https://mlsanity.com/paper/1804.00810/) (2018)
- [Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft](https://mlsanity.com/paper/2003.06066/) (2020)
- [An Optimization Framework for Task Sequencing in Curriculum Learning](https://mlsanity.com/paper/1901.11478/) (2019)
- [Curriculum Learning for Cumulative Return Maximization](https://mlsanity.com/paper/1906.06178/) (2019)
- [ScreenerNet: Learning Self-Paced Curriculum for Deep Neural Networks](https://mlsanity.com/paper/1801.00904/) (2018)
- [Learning Value Functions in Deep Policy Gradients using Residual Variance](https://mlsanity.com/paper/2010.04440/) (2020)

## Built on by (most cited first, 2 of 2)

- [Continual Driving Policy Optimization with Closed-Loop Individualized Curricula](https://mlsanity.com/paper/2309.14209/) (2023)
- [Towards Sample-Efficiency and Generalization of Transfer and Inverse Reinforcement Learning: A Comprehensive Literature Review](https://mlsanity.com/paper/2411.10268/) (2024)

## Abstract (arXiv)

Reinforcement learning (rl) is a popular paradigm for sequential decision making problems. The past decade's advances in rl have led to breakthroughs in many challenging domains such as video games, board games, robotics, and chip design. The sample inefficiency of deep reinforcement learning methods is a significant obstacle when applying rl to real-world problems. Transfer learning has been applied to reinforcement learning such that the knowledge gained in one task can be applied when training in a new task. Curriculum learning is concerned with sequencing tasks or data samples such that knowledge can be transferred between those tasks to learn a target task that would otherwise be too difficult to solve. Designing a curriculum that improves sample efficiency is a complex problem. In this thesis, we propose a teacher-student curriculum learning setting where we simultaneously train a teacher that selects tasks for the student while the student learns how to solve the selected task. Our method is independent of human domain knowledge and manual curriculum design. We evaluated our methods on two reinforcement learning benchmarks: grid world and the challenging Google Football environment. With our method, we can improve the sample efficiency and generality of the student compared to tabula-rasa reinforcement learning.