# Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications

arXiv: https://arxiv.org/abs/1805.07687 · HTML: https://mlsanity.com/paper/1805.07687/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 68 references that are in the map.
- First posted on arXiv 20 May 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 11 of 11)

- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [An Overview of Machine Teaching](https://mlsanity.com/paper/1801.05927/) (2018)
- [Enabling Robots to Communicate their Objectives](https://mlsanity.com/paper/1702.03465/) (2017)
- [Near-Optimally Teaching the Crowd to Classify](https://mlsanity.com/paper/1402.2092/) (2014)
- [The Teaching Dimension of Linear Learners](https://mlsanity.com/paper/1512.02181/) (2015)
- [Machine Teaching for Bayesian Learners in the Exponential Family](https://mlsanity.com/paper/1306.4947/) (2013)
- [Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning](https://mlsanity.com/paper/1707.00724/) (2017)
- [Preference-based Teaching](https://mlsanity.com/paper/1702.02047/) (2017)
- [Exploiting Model Equivalences for Solving Interactive Dynamic Influence Diagrams](https://mlsanity.com/paper/1401.4600/) (2014)

## Built on by (most cited first, 28 of 28)

- [Human-in-the-Loop Imitation Learning using Remote Teleoperation](https://mlsanity.com/paper/2012.06733/) (2020)
- [Policy Poisoning in Batch Reinforcement Learning and Control](https://mlsanity.com/paper/1910.05821/) (2019)
- [Self-Supervised Online Reward Shaping in Sparse-Reward Environments](https://mlsanity.com/paper/2103.04529/) (2021)
- [Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments](https://mlsanity.com/paper/2102.08492/) (2021)
- [Interactive Teaching Algorithms for Inverse Reinforcement Learning](https://mlsanity.com/paper/1905.11867/) (2019)
- [Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints](https://mlsanity.com/paper/1906.00429/) (2019)
- [Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization](https://mlsanity.com/paper/2011.08541/) (2020)
- [Reinforcement Learning Under Algorithmic Triage](https://mlsanity.com/paper/2109.11328/) (2021)
- [Fair Clustering Using Antidote Data](https://mlsanity.com/paper/2106.00600/) (2021)
- [Curriculum Design for Teaching via Demonstrations: Theory and Applications](https://mlsanity.com/paper/2106.04696/) (2021)
- [Interactive Inverse Reinforcement Learning for Cooperative Games](https://mlsanity.com/paper/2111.04698/) (2021)
- [Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations](https://mlsanity.com/paper/1907.03976/) (2019)
- [Learning Data Teaching Strategies Via Knowledge Tracing](https://mlsanity.com/paper/2111.07083/) (2021)
- [Reasoning about Counterfactuals to Improve Human Inverse Reinforcement Learning](https://mlsanity.com/paper/2203.01855/) (2022)
- [Non-Clashing Teaching Maps for Balls in Graphs](https://mlsanity.com/paper/2309.02876/) (2023)
- [The Computational Complexity of Positive Non-Clashing Teaching in Graphs](https://mlsanity.com/paper/2503.07665/) (2025)
- [Dynamically Switching Human Prediction Models for Efficient Planning](https://mlsanity.com/paper/2103.07815/) (2021)
- [Reinforcement Teaching](https://mlsanity.com/paper/2204.11897/) (2022)
- [Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning](https://mlsanity.com/paper/2211.15542/) (2022)
- [Utility-based Adaptive Teaching Strategies using Bayesian Theory of Mind](https://mlsanity.com/paper/2309.17275/) (2023)
- [Inverse Reinforcement Learning for Minimum-Exposure Paths in Spatiotemporally Varying Scalar Fields](https://mlsanity.com/paper/2503.06611/) (2025)
- [Non-Clashing Teaching in Graphs: Algorithms, Complexity, and Bounds](https://mlsanity.com/paper/2602.00657/) (2026)
- [Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning](https://mlsanity.com/paper/2607.08647/) (2026)
- [Interaction-limited Inverse Reinforcement Learning](https://mlsanity.com/paper/2007.00425/) (2020)
- [Address Behaviour Vulnerabilities in the Next Generation of Autonomous Robots](https://mlsanity.com/paper/2103.13268/) (2021)
- [Know Thy Student: Interactive Learning with Gaussian Processes](https://mlsanity.com/paper/2204.12072/) (2022)
- [Interactively Teaching an Inverse Reinforcement Learner with Limited Feedback](https://mlsanity.com/paper/2309.09095/) (2023)
- [Optimally Teaching a Linear Behavior Cloning Agent](https://mlsanity.com/paper/2311.15399/) (2023)

## Abstract (arXiv)

Inverse reinforcement learning (IRL) infers a reward function from demonstrations, allowing for policy improvement and generalization. However, despite much recent interest in IRL, little work has been done to understand the minimum set of demonstrations needed to teach a specific sequential decision-making task. We formalize the problem of finding maximally informative demonstrations for IRL as a machine teaching problem where the goal is to find the minimum number of demonstrations needed to specify the reward equivalence class of the demonstrator. We extend previous work on algorithmic teaching for sequential decision-making tasks by showing a reduction to the set cover problem which enables an efficient approximation algorithm for determining the set of maximally-informative demonstrations. We apply our proposed machine teaching algorithm to two novel applications: providing a lower bound on the number of queries needed to learn a policy using active IRL and developing a novel IRL algorithm that can learn more efficiently from informative demonstrations than a standard IRL approach.