# Human irrationality: both bad and good for reward inference

arXiv: https://arxiv.org/abs/2111.06956 · HTML: https://mlsanity.com/paper/2111.06956/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 12 Nov 2021.

## Builds on (most cited first, 11 of 11)

- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfer](https://mlsanity.com/paper/1609.07088/) (2016)
- [The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities](https://mlsanity.com/paper/1803.03453/) (2018)
- [On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inference](https://mlsanity.com/paper/1906.09624/) (2019)
- [Learning the Preferences of Ignorant, Inconsistent Agents](https://mlsanity.com/paper/1512.05832/) (2015)
- [Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior](https://mlsanity.com/paper/1805.08010/) (2018)
- [Pragmatic-Pedagogic Value Alignment](https://mlsanity.com/paper/1707.06354/) (2017)
- [Task Transfer by Preference-Based Cost Learning](https://mlsanity.com/paper/1805.04686/) (2018)
- [Risk-sensitive Inverse Reinforcement Learning via Semi- and Non-Parametric Methods](https://mlsanity.com/paper/1711.10055/) (2017)

## Built on by (most cited first, 25 of 25)

- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [KTO: Model Alignment as Prospect Theoretic Optimization](https://mlsanity.com/paper/2402.01306/) (2024)
- [Misspecification in Inverse Reinforcement Learning](https://mlsanity.com/paper/2212.03201/) (2022)
- [The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types](https://mlsanity.com/paper/2208.10687/) (2022)
- [On the Sensitivity of Reward Inference to Misspecified Human Models](https://mlsanity.com/paper/2212.04717/) (2022)
- [Active Reward Learning from Multiple Teachers](https://mlsanity.com/paper/2303.00894/) (2023)
- [LLM-driven Imitation of Subrational Behavior : Illusion or Reality?](https://mlsanity.com/paper/2402.08755/) (2024)
- [Learning Preferences for Interactive Autonomy](https://mlsanity.com/paper/2210.10899/) (2022)
- [Concept Alignment as a Prerequisite for Value Alignment](https://mlsanity.com/paper/2310.20059/) (2023)
- [(Ir)rationality in AI: State of the Art, Research Challenges and Open Questions](https://mlsanity.com/paper/2311.17165/) (2023)
- [Zero-Shot Assistance in Sequential Decision Problems](https://mlsanity.com/paper/2202.07364/) (2022)
- [Goal Inference from Open-Ended Dialog](https://mlsanity.com/paper/2410.13957/) (2024)
- [Partial Identifiability and Misspecification in Inverse Reinforcement Learning](https://mlsanity.com/paper/2411.15951/) (2024)
- [Limited or Biased: Modeling Sub-Rational Human Investors in Financial Markets](https://mlsanity.com/paper/2210.08569/) (2022)
- [A behavioural transformer for effective collaboration between a robot and a non-stationary human](https://mlsanity.com/paper/2307.13447/) (2023)
- [Goal Recognition Design for General Behavioral Agents using Machine Learning](https://mlsanity.com/paper/2404.03054/) (2024)
- [Inverse Concave-Utility Reinforcement Learning is Inverse Game Theory](https://mlsanity.com/paper/2405.19024/) (2024)
- [On the Utility of Accounting for Human Beliefs about AI Intention in Human-AI Collaboration](https://mlsanity.com/paper/2406.06051/) (2024)
- [Representation Alignment from Human Feedback for Cross-Embodiment Reward Learning from Mixed-Quality Demonstrations](https://mlsanity.com/paper/2408.05610/) (2024)
- [Supervised Reward Inference](https://mlsanity.com/paper/2502.18447/) (2025)
- [Multi-Agent Inverse Q-Learning from Demonstrations](https://mlsanity.com/paper/2503.04679/) (2025)
- [A Descriptive and Normative Theory of Human Beliefs in RLHF](https://mlsanity.com/paper/2506.01692/) (2025)
- [Humanline: Online Alignment as Perceptual Loss](https://mlsanity.com/paper/2509.24207/) (2025)
- [Mitigating Cognitive Bias in RLHF by Altering Rationality](https://mlsanity.com/paper/2605.06895/) (2026)
- [Learning social norms enhances compatibility in dynamic human-AI coordination](https://mlsanity.com/paper/2607.07021/) (2026)

## Abstract (arXiv)

Assuming humans are (approximately) rational enables robots to infer reward functions by observing human behavior. But people exhibit a wide array of irrationalities, and our goal with this work is to better understand the effect they can have on reward inference. The challenge with studying this effect is that there are many types of irrationality, with varying degrees of mathematical formalization. We thus operationalize irrationality in the language of MDPs, by altering the Bellman optimality equation, and use this framework to study how these alterations would affect inference. We find that wrongly modeling a systematically irrational human as noisy-rational performs a lot worse than correctly capturing these biases -- so much so that it can be better to skip inference altogether and stick to the prior! More importantly, we show that an irrational human, when correctly modelled, can communicate more information about the reward than a perfectly rational human can. That is, if a robot has the correct model of a human's irrationality, it can make an even stronger inference than it ever could if the human were rational. Irrationality fundamentally helps rather than hinder reward inference, but it needs to be correctly accounted for.