mlsanityData to 7 Oct 2026

Teachable Reinforcement Learning via Advice Distillation

arXiv 2203.11197 ↗First posted 19 Mar 2022cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human expert, and learning from intermediate forms of supervision (like binary preferences) is time-consuming and extracts little information from each human intervention. Can we overcome these challenges by building agents that learn from rich, interactive feedback instead? We propose a new supervision paradigm for interactive learning based on "teachable" decision-making systems that learn from structured advice provided by an external teacher. We begin by formalizing a class of human-in-the-loop decision making problems in which multiple forms of teacher-provided advice are available to a learner. We then describe a simple learning algorithm for these problems that first learns to interpret advice, then learns from advice to complete tasks even in the absence of human supervision. In puzzle-solving, navigation, and locomotion domains, we show that agents that learn from advice can acquire new skills with significantly less human supervision than standard reinforcement learning algorithms and often less than imitation learning.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 64 references that are in the map. First posted on arXiv 19 Mar 2022.

Builds on

31 of 64 references.

References · 31

  1. 2017Deep reinforcement learning from human preferencescited by 3,846
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  4. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  5. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  8. 2016Cooperative Inverse Reinforcement Learningcited by 298
  9. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  10. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 238
  11. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  12. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
19 more
  1. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  2. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  3. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  4. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  5. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  6. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  7. 2017Learning with Latent Languagecited by 75
  8. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedbackcited by 56
  9. 2015Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action Sequencescited by 50
  10. 2019Leveraging Human Guidance for Deep Reinforcement Learning Taskscited by 42
  11. 2021A Survey on Interactive Reinforcement Learning: Design Principles and Open Challengescited by 41
  12. 2018Guiding Policies with Language via Meta-Learningcited by 34
  13. 2019Hierarchical Decision Making by Generating and Following Natural Language Instructionscited by 30
  14. 2020Hierarchically Decoupled Imitation for Morphological Transfercited by 30
  15. 2021Interactive Learning from Activity Descriptioncited by 26
  16. 2017Agent-Agnostic Human-in-the-Loop Reinforcement Learningcited by 23
  17. 2019Conditional Driving from Natural Language Instructionscited by 17
  18. 2017Grounding Language for Transfer in Deep Reinforcement Learningcited by 5
  19. 2017A Tale of Two DRAGGNs: A Hybrid Approach for Interpreting Action-Oriented and Goal-Oriented Instructionscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023Continual Robot Learning using Self-Supervised Task Inferencecited by 5
  2. 2024Policy Learning with a Language Bottleneckcited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2026RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning
  2. 2019Learning to Interactively Learn and Assist
  3. 2020Bridging the Imitation Gap by Adaptive Insubordination
  4. 2022The Expertise Problem: Learning from Specialized Feedback
  5. 2020Human AI interaction loop training: New approach for interactive reinforcement learning
  6. 2025CRAFT: Coaching Reinforcement Learning Autonomously using Foundation Models for Multi-Robot Coordination Tasks
  7. 2025Distilling Realizable Students from Unrealizable Teachers
  8. 2026Agentic Reinforcement Learning with Self-Distilled Reward Shaping