mlsanityData to 7 Oct 2026

Automatic Curricula via Expert Demonstrations

arXiv 2106.09159 ↗First posted 16 Jun 2021cs.LG cs.ROcited by 4

On the map: Offline RL and imitation learning from data → Real-world robot offline RL

Abstract (arXiv)

We propose Automatic Curricula via Expert Demonstrations (ACED), a reinforcement learning (RL) approach that combines the ideas of imitation learning and curriculum learning in order to solve challenging robotic manipulation tasks with sparse reward functions. Curriculum learning solves complicated RL tasks by introducing a sequence of auxiliary tasks with increasing difficulty, yet how to automatically design effective and generalizable curricula remains a challenging research problem. ACED extracts curricula from a small amount of expert demonstration trajectories by dividing demonstrations into sections and initializing training episodes to states sampled from different sections of demonstrations. Through moving the reset states from the end to the beginning of demonstrations as the learning agent improves its performance, ACED not only learns challenging manipulation tasks with unseen initializations and goals, but also discovers novel solutions that are distinct from the demonstrations. In addition, ACED can be naturally combined with other imitation learning methods to utilize expert demonstrations in a more efficient manner, and we show that a combination of ACED with behavior cloning allows pick-and-place tasks to be learned with as few as 1 demonstration and block stacking tasks to be learned with 20 demonstrations.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Real-world robot offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 41 references that are in the map. First posted on arXiv 16 Jun 2021.

Builds on

31 of 41 references.

References · 31

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  7. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  8. 2016The Option-Critic Architecturecited by 433
  9. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  10. 2018ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worstcited by 391
  11. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
19 more
  1. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  2. 2017Automated Curriculum Learning for Neural Networkscited by 251
  3. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  4. 2019Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutionscited by 182
  5. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  6. 2018Learning to Navigate in Cities Without a Mapcited by 103
  7. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103
  8. 2017Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-To-End Learning from Demonstrationcited by 100
  9. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  10. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  11. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 76
  12. 2020Safe Reinforcement Learning via Curriculum Inductioncited by 73
  13. 2018Latent Space Policies for Hierarchical Reinforcement Learningcited by 66
  14. 2018Deep Object-Centric Policies for Autonomous Drivingcited by 41
  15. 2020Continual Learning of Control Primitives: Skill Discovery via Reset-Gamescited by 29
  16. 2018BaRC: Backward Reachability Curriculum for Robotic Reinforcement Learningcited by 21
  17. 2018Backplay: "Man muss immer umkehren"cited by 16
  18. 2019Curiosity-Driven Multi-Criteria Hindsight Experience Replaycited by 11
  19. 2020An Empowerment-based Solution to Robotic Manipulation Tasks with Sparse Rewardscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 0; 2026: 2.

4 papers.

  1. 2023Imitation Learning from Observation with Automatic Discount Schedulingcited by 10
  2. 2022Case-Based Inverse Reinforcement Learning Using Temporal Coherencecited by 1
  3. 2026Learning from Demonstrations via Capability-Aware Goal Samplingcited by 1
  4. 2026SCOUT: Per-Context Reset Curricula for Sparse-Reward Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Task Phasing: Automated Curriculum Learning from Demonstrations
  2. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward
  3. 2023A Survey of Demonstration Learning
  4. 2026Decompose and Reorganize: Planning with Primitives and Visuomotor Policies Learned from Demonstrations ↗
  5. 2021Curriculum Design for Teaching via Demonstrations: Theory and Applications
  6. 2024Learning Manipulation Skills through Robot Chain-of-Thought with Sparse Failure Guidance
  7. 2026PRISM: Personalized Refinement of Imitation Skills for Manipulation via Human Instructions
  8. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations