mlsanityData to 14 Oct 2026

ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations

arXiv 1910.12154 ↗First posted 26 Oct 2019cs.LG cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

Learning from demonstrations is a popular tool for accelerating and reducing the exploration requirements of reinforcement learning. When providing expert demonstrations to human students, we know that the demonstrations must fall within a particular range of difficulties called the "Zone of Proximal Development (ZPD)". If they are too easy the student learns nothing, but if they are too difficult the student is unable to follow along. This raises the question: Given a set of potential demonstrators, which among them is best suited for teaching any particular learner? Prior work, such as the popular Deep Q-learning from Demonstrations (DQfD) algorithm has generally focused on single demonstrators. In this work we consider the problem of choosing among multiple demonstrators of varying skill levels. Our results align with intuition from human learners: it is not always the best policy to draw demonstrations from the best performing demonstrator (in terms of reward). We show that careful selection of teaching strategies can result in sample efficiency gains in the learner's environment across nine Atari games

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 24 of its 34 references that are in the map. First posted on arXiv 26 Oct 2019.

Builds on

24 of 34 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2016OpenAI Gymcited by 1,736
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2015Prioritized Experience Replaycited by 1,099
  8. 2018Exploration by Random Network Distillationcited by 993
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  11. 2018Born Again Neural Networkscited by 546
  12. 2015Policy Distillationcited by 440
12 more
  1. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  2. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  3. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  4. 2018Distributed Prioritized Experience Replaycited by 269
  5. 2017A Deeper Look at Experience Replaycited by 105
  6. 2018Kickstarting Deep Reinforcement Learningcited by 81
  7. 2018Learning Montezuma's Revenge from a Single Demonstrationcited by 80
  8. 2019Diagnosing Bottlenecks in Deep Q-learning Algorithmscited by 73
  9. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  10. 2018Observe and Look Further: Achieving Consistent Performance on Ataricited by 44
  11. 2019An Optimistic Perspective on Offline Reinforcement Learningcited by 37
  12. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 36

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 0; 2023: 0; 2024: 1; 2025: 2.

5 papers.

  1. 2025Shared Autonomy for Proximal Teachingcited by 4
  2. 2021Towards Teachable Autotelic Agentscited by 2
  3. 2024GLIDE-RL: Grounded Language Instruction through DEmonstration in RLcited by 2
  4. 2021DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learningcited by 0
  5. 2025Autonomous Curriculum Design via Relative Entropy Based Task Modificationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Curriculum Design for Teaching via Demonstrations: Theory and Applications
  2. 2022Bayesian Q-learning With Imperfect Expert Demonstrations
  3. 2025Beyond-Expert Performance with Limited Demonstrations: Efficient Imitation Learning with Double Exploration
  4. 2018Teaching Inverse Reinforcement Learners via Features and Demonstrations
  5. 2018Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications
  6. 2022Self-Imitation Learning from Demonstrations
  7. 2020Demonstration-efficient Inverse Reinforcement Learning in Procedurally Generated Environments
  8. 2018Reward learning from human preferences and demonstrations in Atari