# On the Effect of Robot Errors on Human Teaching Dynamics

arXiv: https://arxiv.org/abs/2409.09827 · HTML: https://mlsanity.com/paper/2409.09827/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 57 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Builds on (most cited first, 13 of 13)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [Reward-rational (implicit) choice: A unifying formalism for reward learning](https://mlsanity.com/paper/2002.04833/) (2020)
- [Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferences](https://mlsanity.com/paper/2006.14091/) (2020)
- [The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types](https://mlsanity.com/paper/2208.10687/) (2022)
- [Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learning](https://mlsanity.com/paper/2206.13316/) (2022)
- [Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback](https://mlsanity.com/paper/2402.02423/) (2024)
- [FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback](https://mlsanity.com/paper/2001.06781/) (2020)
- [RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback](https://mlsanity.com/paper/2308.04332/) (2023)
- [Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning](https://mlsanity.com/paper/1807.09991/) (2018)
- [Improving User Specifications for Robot Behavior through Active Preference Learning: Framework and Evaluation](https://mlsanity.com/paper/1907.10412/) (2019)

## Abstract (arXiv)

Human-in-the-loop learning is gaining popularity, particularly in the field of robotics, because it leverages human knowledge about real-world tasks to facilitate agent learning. When people instruct robots, they naturally adapt their teaching behavior in response to changes in robot performance. While current research predominantly focuses on integrating human teaching dynamics from an algorithmic perspective, understanding these dynamics from a human-centered standpoint is an under-explored, yet fundamental problem. Addressing this issue will enhance both robot learning and user experience. Therefore, this paper explores one potential factor contributing to the dynamic nature of human teaching: robot errors. We conducted a user study to investigate how the presence and severity of robot errors affect three dimensions of human teaching dynamics: feedback granularity, feedback richness, and teaching time, in both forced-choice and open-ended teaching contexts. The results show that people tend to spend more time teaching robots with errors, provide more detailed feedback over specific segments of a robot's trajectory, and that robot error can influence a teacher's choice of feedback modality. Our findings offer valuable insights for designing effective interfaces for interactive learning and optimizing algorithms to better understand human intentions.