# Interactive Teaching Algorithms for Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/1905.11867 · HTML: https://mlsanity.com/paper/1905.11867/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 17 of its 60 references that are in the map.
- First posted on arXiv 28 May 2019.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 17 of 17)

- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization](https://mlsanity.com/paper/1603.00448/) (2016)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1507.04888/) (2015)
- [An Overview of Machine Teaching](https://mlsanity.com/paper/1801.05927/) (2018)
- [Iterative Machine Teaching](https://mlsanity.com/paper/1705.10470/) (2017)
- [First-Person Activity Forecasting with Online Inverse Reinforcement Learning](https://mlsanity.com/paper/1612.07796/) (2016)
- [Near-Optimally Teaching the Crowd to Classify](https://mlsanity.com/paper/1402.2092/) (2014)
- [Repeated Inverse Reinforcement Learning](https://mlsanity.com/paper/1705.05427/) (2017)
- [Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications](https://mlsanity.com/paper/1805.07687/) (2018)
- [Dual Policy Iteration](https://mlsanity.com/paper/1805.10755/) (2018)
- [Towards Black-box Iterative Machine Teaching](https://mlsanity.com/paper/1710.07742/) (2017)
- [Teaching Inverse Reinforcement Learners via Features and Demonstrations](https://mlsanity.com/paper/1810.08926/) (2018)
- [Teaching Multiple Concepts to a Forgetful Learner](https://mlsanity.com/paper/1805.08322/) (2018)
- [Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners](https://mlsanity.com/paper/1802.05190/) (2018)
- [Iterative Classroom Teaching](https://mlsanity.com/paper/1811.03537/) (2018)

## Built on by (most cited first, 24 of 24)

- [Policy Poisoning in Batch Reinforcement Learning and Control](https://mlsanity.com/paper/1910.05821/) (2019)
- [Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch](https://mlsanity.com/paper/2007.01174/) (2020)
- [Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments](https://mlsanity.com/paper/2102.08492/) (2021)
- [Reinforcement Learning for Education: Opportunities and Challenges](https://mlsanity.com/paper/2107.08828/) (2021)
- [Proximal Curriculum for Reinforcement Learning Agents](https://mlsanity.com/paper/2304.12877/) (2023)
- [Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints](https://mlsanity.com/paper/1906.00429/) (2019)
- [Reinforcement Learning Under Algorithmic Triage](https://mlsanity.com/paper/2109.11328/) (2021)
- [Teaching an Active Learner with Contrastive Examples](https://mlsanity.com/paper/2110.14888/) (2021)
- [Fair Clustering Using Antidote Data](https://mlsanity.com/paper/2106.00600/) (2021)
- [Iterative Teaching by Label Synthesis](https://mlsanity.com/paper/2110.14432/) (2021)
- [Curriculum Design for Teaching via Demonstrations: Theory and Applications](https://mlsanity.com/paper/2106.04696/) (2021)
- [Nonparametric Iterative Machine Teaching](https://mlsanity.com/paper/2306.03007/) (2023)
- [Iterative Teaching by Data Hallucination](https://mlsanity.com/paper/2210.17467/) (2022)
- [Interactive Inverse Reinforcement Learning for Cooperative Games](https://mlsanity.com/paper/2111.04698/) (2021)
- [Nonparametric Teaching for Multiple Learners](https://mlsanity.com/paper/2311.10318/) (2023)
- [LLM-Mediated Guidance of MARL Systems](https://mlsanity.com/paper/2503.13553/) (2025)
- [Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2211.15542/) (2022)
- [Active Third-Person Imitation Learning](https://mlsanity.com/paper/2312.16365/) (2023)
- [Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning](https://mlsanity.com/paper/2607.08647/) (2026)
- [Interaction-limited Inverse Reinforcement Learning](https://mlsanity.com/paper/2007.00425/) (2020)
- [One-shot Machine Teaching: Cost Very Few Examples to Converge Faster](https://mlsanity.com/paper/2212.06416/) (2022)
- [Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback](https://mlsanity.com/paper/2309.09095/) (2023)
- [Optimally Teaching a Linear Behavior Cloning Agent](https://mlsanity.com/paper/2311.15399/) (2023)
- [Interactive Inverse Reinforcement Learning of Interaction Scenarios via Bi-level Optimization](https://mlsanity.com/paper/2605.08131/) (2026)

## Abstract (arXiv)

We study the problem of inverse reinforcement learning (IRL) with the added twist that the learner is assisted by a helpful teacher. More formally, we tackle the following algorithmic question: How could a teacher provide an informative sequence of demonstrations to an IRL learner to speed up the learning process? We present an interactive teaching framework where a teacher adaptively chooses the next demonstration based on learner's current policy. In particular, we design teaching algorithms for two concrete settings: an omniscient setting where a teacher has full knowledge about the learner's dynamics and a blackbox setting where the teacher has minimal knowledge. Then, we study a sequential variant of the popular MCE-IRL learner and prove convergence guarantees of our teaching algorithm in the omniscient setting. Extensive experiments with a car driving simulator environment show that the learning progress can be speeded up drastically as compared to an uninformative teacher.