# Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints

arXiv: https://arxiv.org/abs/1906.00429 · HTML: https://mlsanity.com/paper/1906.00429/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 19 of its 52 references that are in the map.
- First posted on arXiv 2 Jun 2019.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 19 of 19)

- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Multi-agent Reinforcement Learning in Sequential Social Dilemmas](https://mlsanity.com/paper/1702.03037/) (2017)
- [A Review on Bilevel Optimization: From Classical to Evolutionary Approaches and Applications](https://mlsanity.com/paper/1705.06270/) (2017)
- [An Overview of Machine Teaching](https://mlsanity.com/paper/1801.05927/) (2018)
- [Near-Optimally Teaching the Crowd to Classify](https://mlsanity.com/paper/1402.2092/) (2014)
- [Teaching Categories to Human Learners with Visual Explanations](https://mlsanity.com/paper/1802.06924/) (2018)
- [Repeated Inverse Reinforcement Learning](https://mlsanity.com/paper/1705.05427/) (2017)
- [Machine Teaching for Inverse Reinforcement Learning: Algorithms and Applications](https://mlsanity.com/paper/1805.07687/) (2018)
- [Risk-Aware Active Inverse Reinforcement Learning](https://mlsanity.com/paper/1901.02161/) (2019)
- [Interactive Teaching Algorithms for Inverse Reinforcement Learning](https://mlsanity.com/paper/1905.11867/) (2019)
- [Towards Black-box Iterative Machine Teaching](https://mlsanity.com/paper/1710.07742/) (2017)
- [The Teaching Dimension of Linear Learners](https://mlsanity.com/paper/1512.02181/) (2015)
- [Teaching Inverse Reinforcement Learners via Features and Demonstrations](https://mlsanity.com/paper/1810.08926/) (2018)
- [Machine Teaching for Bayesian Learners in the Exponential Family](https://mlsanity.com/paper/1306.4947/) (2013)
- [Efficient Probabilistic Performance Bounds for Inverse Reinforcement Learning](https://mlsanity.com/paper/1707.00724/) (2017)
- [Teaching Multiple Concepts to a Forgetful Learner](https://mlsanity.com/paper/1805.08322/) (2018)
- [Understanding the Role of Adaptivity in Machine Teaching: The Case of Version Space Learners](https://mlsanity.com/paper/1802.05190/) (2018)
- [Iterative Classroom Teaching](https://mlsanity.com/paper/1811.03537/) (2018)

## Built on by (most cited first, 18 of 18)

- [Investigating Bi-Level Optimization for Learning and Vision from a Unified Perspective: A Survey and Beyond](https://mlsanity.com/paper/2101.11517/) (2021)
- [A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse](https://mlsanity.com/paper/2112.04660/) (2021)
- [Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch](https://mlsanity.com/paper/2007.01174/) (2020)
- [MEDIRL: Predicting the Visual Attention of Drivers via Maximum Entropy Deep Inverse Reinforcement Learning](https://mlsanity.com/paper/1912.07773/) (2019)
- [Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments](https://mlsanity.com/paper/2102.08492/) (2021)
- [Reinforcement Learning for Education: Opportunities and Challenges](https://mlsanity.com/paper/2107.08828/) (2021)
- [Local Stochastic Bilevel Optimization with Momentum-Based Variance Reduction](https://mlsanity.com/paper/2205.01608/) (2022)
- [Identifiability and Generalizability in Constrained Inverse Reinforcement Learning](https://mlsanity.com/paper/2306.00629/) (2023)
- [Reinforcement Learning Under Algorithmic Triage](https://mlsanity.com/paper/2109.11328/) (2021)
- [Teaching an Active Learner with Contrastive Examples](https://mlsanity.com/paper/2110.14888/) (2021)
- [Fair Clustering Using Antidote Data](https://mlsanity.com/paper/2106.00600/) (2021)
- [Iterative Teaching by Label Synthesis](https://mlsanity.com/paper/2110.14432/) (2021)
- [Curriculum Design for Teaching via Demonstrations: Theory and Applications](https://mlsanity.com/paper/2106.04696/) (2021)
- [Iterative Teaching by Data Hallucination](https://mlsanity.com/paper/2210.17467/) (2022)
- [Fine-grained Analysis of Stability and Generalization for Stochastic Bilevel Optimization](https://mlsanity.com/paper/2604.04090/) (2026)
- [On Stability and Generalization of Bilevel Optimization Problem](https://mlsanity.com/paper/2210.01063/) (2022)
- [Multi-Modal, Multi-Environment Machine Teaching for Robust Reward Learning](https://mlsanity.com/paper/2607.08647/) (2026)
- [Optimally Teaching a Linear Behavior Cloning Agent](https://mlsanity.com/paper/2311.15399/) (2023)

## Abstract (arXiv)

Inverse reinforcement learning (IRL) enables an agent to learn complex behavior by observing demonstrations from a (near-)optimal policy. The typical assumption is that the learner's goal is to match the teacher's demonstrated behavior. In this paper, we consider the setting where the learner has its own preferences that it additionally takes into consideration. These preferences can for example capture behavioral biases, mismatched worldviews, or physical constraints. We study two teaching approaches: learner-agnostic teaching, where the teacher provides demonstrations from an optimal policy ignoring the learner's preferences, and learner-aware teaching, where the teacher accounts for the learner's preferences. We design learner-aware teaching algorithms and show that significant performance improvements can be achieved over learner-agnostic teaching.