# Using Machine Teaching to Investigate Human Assumptions when Teaching Reinforcement Learners

arXiv: https://arxiv.org/abs/2009.02476 · HTML: https://mlsanity.com/paper/2009.02476/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Sep 2020.

## Built on by (most cited first, 3 of 3)

- [Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework](https://mlsanity.com/paper/2411.11761/) (2024)
- [One-shot Machine Teaching: Cost Very Few Examples to Converge Faster](https://mlsanity.com/paper/2212.06416/) (2022)
- [The Delusional Hedge Algorithm as a Model of Human Learning from Diverse Opinions](https://mlsanity.com/paper/2402.13927/) (2024)

## Abstract (arXiv)

Successful teaching requires an assumption of how the learner learns - how the learner uses experiences from the world to update their internal states. We investigate what expectations people have about a learner when they teach them in an online manner using rewards and punishment. We focus on a common reinforcement learning method, Q-learning, and examine what assumptions people have using a behavioral experiment. To do so, we first establish a normative standard, by formulating the problem as a machine teaching optimization problem. To solve the machine teaching optimization problem, we use a deep learning approximation method which simulates learners in the environment and learns to predict how feedback affects the learner's internal states. What do people assume about a learner's learning and discount rates when they teach them an idealized exploration-exploitation task? In a behavioral experiment, we find that people can teach the task to Q-learners in a relatively efficient and effective manner when the learner uses a small value for its discounting rate and a large value for its learning rate. However, they still are suboptimal. We also find that providing people with real-time updates of how possible feedback would affect the Q-learner's internal states weakly helps them teach. Our results reveal how people teach using evaluative feedback and provide guidance for how engineers should design machine agents in a manner that is intuitive for people.