# Automatic Curricula via Expert Demonstrations

arXiv: https://arxiv.org/abs/2106.09159 · HTML: https://mlsanity.com/paper/2106.09159/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 31 of its 41 references that are in the map.
- First posted on arXiv 16 Jun 2021.

## Placed in

- Offline RL and imitation learning from data / [Real-world robot offline RL](https://mlsanity.com/offlinerl/real-world-robot-offline-rl/)

## Builds on (most cited first, 30 of 31)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst](https://mlsanity.com/paper/1812.03079/) (2018)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards](https://mlsanity.com/paper/1707.08817/) (2017)
- [Automatic Goal Generation for Reinforcement Learning Agents](https://mlsanity.com/paper/1705.06366/) (2017)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [Reverse Curriculum Generation for Reinforcement Learning](https://mlsanity.com/paper/1707.05300/) (2017)
- [Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions](https://mlsanity.com/paper/1901.01753/) (2019)
- [Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play](https://mlsanity.com/paper/1703.05407/) (2017)
- [Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow](https://mlsanity.com/paper/1810.00821/) (2018)
- [Learning to Navigate in Cities Without a Map](https://mlsanity.com/paper/1804.00168/) (2018)
- [Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstration](https://mlsanity.com/paper/1707.02920/) (2017)
- [Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning](https://mlsanity.com/paper/1711.06782/) (2017)
- [Learning Montezuma's Revenge from a Single Demonstration](https://mlsanity.com/paper/1812.03381/) (2018)
- [CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning](https://mlsanity.com/paper/1810.06284/) (2018)
- [Safe Reinforcement Learning via Curriculum Induction](https://mlsanity.com/paper/2006.12136/) (2020)
- [Latent Space Policies for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1804.02808/) (2018)
- [Deep Object-Centric Policies for Autonomous Driving](https://mlsanity.com/paper/1811.05432/) (2018)
- [Continual Learning of Control Primitives: Skill Discovery via Reset-Games](https://mlsanity.com/paper/2011.05286/) (2020)
- [BaRC: Backward Reachability Curriculum for Robotic Reinforcement Learning](https://mlsanity.com/paper/1806.06161/) (2018)
- [Backplay: "Man muss immer umkehren"](https://mlsanity.com/paper/1807.06919/) (2018)
- [Curiosity-Driven Multi-Criteria Hindsight Experience Replay](https://mlsanity.com/paper/1906.03710/) (2019)

## Built on by (most cited first, 5 of 5)

- [Imitation Learning from Observation with Automatic Discount Scheduling](https://mlsanity.com/paper/2310.07433/) (2023)
- [Case-Based Inverse Reinforcement Learning Using Temporal Coherence](https://mlsanity.com/paper/2206.05827/) (2022)
- [Learning from Demonstrations via Capability-Aware Goal Sampling](https://mlsanity.com/paper/2601.08731/) (2026)
- [Per-Context Reverse Curricula for Sparse-Reward Reinforcement Learning](https://mlsanity.com/paper/2607.26417/) (2026)
- [DexMimic: A Generalist Dexterous Hand-Object Tracking Policy via Value-Guided Reset Allocation](https://mlsanity.com/paper/2610.19089/) (2026)

## Abstract (arXiv)

We propose Automatic Curricula via Expert Demonstrations (ACED), a reinforcement learning (RL) approach that combines the ideas of imitation learning and curriculum learning in order to solve challenging robotic manipulation tasks with sparse reward functions. Curriculum learning solves complicated RL tasks by introducing a sequence of auxiliary tasks with increasing difficulty, yet how to automatically design effective and generalizable curricula remains a challenging research problem. ACED extracts curricula from a small amount of expert demonstration trajectories by dividing demonstrations into sections and initializing training episodes to states sampled from different sections of demonstrations. Through moving the reset states from the end to the beginning of demonstrations as the learning agent improves its performance, ACED not only learns challenging manipulation tasks with unseen initializations and goals, but also discovers novel solutions that are distinct from the demonstrations. In addition, ACED can be naturally combined with other imitation learning methods to utilize expert demonstrations in a more efficient manner, and we show that a combination of ACED with behavior cloning allows pick-and-place tasks to be learned with as few as 1 demonstration and block stacking tasks to be learned with 20 demonstrations.