# When Humans Aren't Optimal: Robots that Collaborate with Risk-Aware Humans

arXiv: https://arxiv.org/abs/2001.04377 · HTML: https://mlsanity.com/paper/2001.04377/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jan 2020.

## Built on by (most cited first, 30 of 34)

- [KTO: Model Alignment as Prospect Theoretic Optimization](https://mlsanity.com/paper/2402.01306/) (2024)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [Inverse Preference Learning: Preference-based RL without a Reward Function](https://mlsanity.com/paper/2305.15363/) (2023)
- [Learning Multimodal Rewards from Rankings](https://mlsanity.com/paper/2109.12750/) (2021)
- [Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased](https://mlsanity.com/paper/2302.01605/) (2023)
- [Learning from Imperfect Demonstrations from Agents with Varying Dynamics](https://mlsanity.com/paper/2103.05910/) (2021)
- [A Ranking Game for Imitation Learning](https://mlsanity.com/paper/2202.03481/) (2022)
- [Algorithmic Assistance with Recommendation-Dependent Preferences](https://mlsanity.com/paper/2208.07626/) (2022)
- [RAP: Risk-Aware Prediction for Robust Planning](https://mlsanity.com/paper/2210.01368/) (2022)
- [Conformalized Interactive Imitation Learning: Handling Expert Shift and Intermittent Feedback](https://mlsanity.com/paper/2410.08852/) (2024)
- [Deception Game: Closing the Safety-Learning Loop in Interactive Robot Autonomy](https://mlsanity.com/paper/2309.01267/) (2023)
- [MILE: Model-based Intervention Learning](https://mlsanity.com/paper/2502.13519/) (2025)
- [Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2203.04421/) (2022)
- [Batch Active Learning of Reward Functions from Human Preferences](https://mlsanity.com/paper/2402.15757/) (2024)
- [MIRROR: Differentiable Deep Social Projection for Assistive Human-Robot Communication](https://mlsanity.com/paper/2203.02877/) (2022)
- [Game-Theoretic Planning for Autonomous Driving among Risk-Aware Human Drivers](https://mlsanity.com/paper/2205.00562/) (2022)
- [Learning from Imperfect Demonstrations via Adversarial Confidence Transfer](https://mlsanity.com/paper/2202.02967/) (2022)
- [Approximating Discontinuous Nash Equilibrial Values of Two-Player General-Sum Differential Games](https://mlsanity.com/paper/2207.01773/) (2022)
- [Concept Alignment as a Prerequisite for Value Alignment](https://mlsanity.com/paper/2310.20059/) (2023)
- [(Ir)rationality in AI: State of the Art, Research Challenges and Open Questions](https://mlsanity.com/paper/2311.17165/) (2023)
- [Sketch-based Creativity Support Tools using Deep Learning](https://mlsanity.com/paper/2111.09991/) (2021)
- [Value Approximation for Two-Player General-Sum Differential Games with State Constraints](https://mlsanity.com/paper/2311.16520/) (2023)
- [GAN-Based Interactive Reinforcement Learning from Demonstration and Human Evaluative Feedback](https://mlsanity.com/paper/2104.06600/) (2021)
- [Collaborative Decision Making Using Action Suggestions](https://mlsanity.com/paper/2209.13160/) (2022)
- [Robot Navigation in Risky, Crowded Environments: Understanding Human Preferences](https://mlsanity.com/paper/2303.08284/) (2023)
- [Mixed-Initiative Human-Robot Teaming under Suboptimality with Online Bayesian Adaptation](https://mlsanity.com/paper/2403.16178/) (2024)
- [Goal Recognition Design for General Behavioral Agents using Machine Learning](https://mlsanity.com/paper/2404.03054/) (2024)
- [Situated, Dynamic, and Subjective: Envisioning the Design of Theory-of-Mind-Enabled Everyday AI with Industry Practitioners](https://mlsanity.com/paper/2602.11342/) (2026)
- [Visual Intelligence through Human Interaction](https://mlsanity.com/paper/2111.06913/) (2021)
- [Revealed Multi-Objective Utility Aggregation in Human Driving](https://mlsanity.com/paper/2303.07435/) (2023)

## Abstract (arXiv)

In order to collaborate safely and efficiently, robots need to anticipate how their human partners will behave. Some of today's robots model humans as if they were also robots, and assume users are always optimal. Other robots account for human limitations, and relax this assumption so that the human is noisily rational. Both of these models make sense when the human receives deterministic rewards: i.e., gaining either 100 or 130 with certainty. But in real world scenarios, rewards are rarely deterministic. Instead, we must make choices subject to risk and uncertainty--and in these settings, humans exhibit a cognitive bias towards suboptimal behavior. For example, when deciding between gaining 100 with certainty or 130 only 80% of the time, people tend to make the risk-averse choice--even though it leads to a lower expected gain! In this paper, we adopt a well-known Risk-Aware human model from behavioral economics called Cumulative Prospect Theory and enable robots to leverage this model during human-robot interaction (HRI). In our user studies, we offer supporting evidence that the Risk-Aware model more accurately predicts suboptimal human behavior. We find that this increased modeling accuracy results in safer and more efficient human-robot collaboration. Overall, we extend existing rational human models so that collaborative robots can anticipate and plan around suboptimal human behavior during HRI.