# Teaching Inverse Reinforcement Learners via Features and Demonstrations

arXiv: https://arxiv.org/abs/1810.08926 · HTML: https://mlsanity.com/paper/1810.08926/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- First posted on arXiv 21 Oct 2018.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Built on by (most cited first, 19 of 19)

- [Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch](https://mlsanity.com/paper/2007.01174/) (2020)
- [Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments](https://mlsanity.com/paper/2102.08492/) (2021)
- [Reinforcement Learning for Education: Opportunities and Challenges](https://mlsanity.com/paper/2107.08828/) (2021)
- [Interactive Teaching Algorithms for Inverse Reinforcement Learning](https://mlsanity.com/paper/1905.11867/) (2019)
- [Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints](https://mlsanity.com/paper/1906.00429/) (2019)
- [Inducing Structure in Reward Learning by Learning Features](https://mlsanity.com/paper/2201.07082/) (2022)
- [Reinforcement Learning Under Algorithmic Triage](https://mlsanity.com/paper/2109.11328/) (2021)
- [Teaching an Active Learner with Contrastive Examples](https://mlsanity.com/paper/2110.14888/) (2021)
- [Fair Clustering Using Antidote Data](https://mlsanity.com/paper/2106.00600/) (2021)
- [Iterative Teaching by Label Synthesis](https://mlsanity.com/paper/2110.14432/) (2021)
- [Curriculum Design for Teaching via Demonstrations: Theory and Applications](https://mlsanity.com/paper/2106.04696/) (2021)
- [Iterative Teaching by Data Hallucination](https://mlsanity.com/paper/2210.17467/) (2022)
- [Iterative Teacher-Aware Learning](https://mlsanity.com/paper/2110.00137/) (2021)
- [Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method](https://mlsanity.com/paper/2203.04951/) (2022)
- [Active Third-Person Imitation Learning](https://mlsanity.com/paper/2312.16365/) (2023)
- [Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning](https://mlsanity.com/paper/2607.08647/) (2026)
- [Optimally Teaching a Linear Behavior Cloning Agent](https://mlsanity.com/paper/2311.15399/) (2023)
- [Model Selection for Inverse Reinforcement Learning via Structural Risk Minimization](https://mlsanity.com/paper/2312.16566/) (2023)
- [Towards User-Centred Design of AI-Assisted Decision-Making in Law Enforcement](https://mlsanity.com/paper/2504.17393/) (2025)

## Abstract (arXiv)

Learning near-optimal behaviour from an expert's demonstrations typically relies on the assumption that the learner knows the features that the true reward function depends on. In this paper, we study the problem of learning from demonstrations in the setting where this is not the case, i.e., where there is a mismatch between the worldviews of the learner and the expert. We introduce a natural quantity, the teaching risk, which measures the potential suboptimality of policies that look optimal to the learner in this setting. We show that bounds on the teaching risk guarantee that the learner is able to find a near-optimal policy using standard algorithms based on inverse reinforcement learning. Based on these findings, we suggest a teaching scheme in which the expert can decrease the teaching risk by updating the learner's worldview, and thus ultimately enable her to find a near-optimal policy.