# On the Perception of Difficulty: Differences between Humans and AI

arXiv: https://arxiv.org/abs/2304.09803 · HTML: https://mlsanity.com/paper/2304.09803/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 40 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 17 of 17)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance](https://mlsanity.com/paper/2006.14779/) (2020)
- [Understanding Dataset Difficulty with V-Usable Information](https://mlsanity.com/paper/2110.08420/) (2021)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [Partial success in closing the gap between human and machine vision](https://mlsanity.com/paper/2106.07411/) (2021)
- [A Theory of Usable Information Under Computational Constraints](https://mlsanity.com/paper/2002.10689/) (2020)
- [Asking Easy Questions: A User-Friendly Approach to Active Reward Learning](https://mlsanity.com/paper/1910.04365/) (2019)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Trustworthy AI](https://mlsanity.com/paper/2011.02272/) (2020)
- [Ask Not What AI Can Do, But What AI Should Do: Towards a Framework of Task Delegability](https://mlsanity.com/paper/1902.03245/) (2019)
- [Near-Optimally Teaching the Crowd to Classify](https://mlsanity.com/paper/1402.2092/) (2014)
- [Learn like a Pathologist: Curriculum Learning by Annotator Agreement for Histopathology Image Classification](https://mlsanity.com/paper/2009.13698/) (2020)
- [On the Influence of Explainable AI on Automation Bias](https://mlsanity.com/paper/2204.08859/) (2022)
- [On the Effect of Information Asymmetry in Human-AI Teams](https://mlsanity.com/paper/2205.01467/) (2022)
- [Curriculum Learning for Speech Emotion Recognition from Crowdsourced Labels](https://mlsanity.com/paper/1805.10339/) (2018)
- [Training Novices: The Role of Human-AI Collaboration and Knowledge Transfer](https://mlsanity.com/paper/2207.00497/) (2022)
- [On the Influence of Cognitive Styles on Users' Understanding of Explanations](https://mlsanity.com/paper/2210.02123/) (2022)

## Built on by (most cited first, 1 of 1)

- [Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration](https://mlsanity.com/paper/2412.15701/) (2024)

## Abstract (arXiv)

With the increased adoption of artificial intelligence (AI) in industry and society, effective human-AI interaction systems are becoming increasingly important. A central challenge in the interaction of humans with AI is the estimation of difficulty for human and AI agents for single task instances.These estimations are crucial to evaluate each agent's capabilities and, thus, required to facilitate effective collaboration. So far, research in the field of human-AI interaction estimates the perceived difficulty of humans and AI independently from each other. However, the effective interaction of human and AI agents depends on metrics that accurately reflect each agent's perceived difficulty in achieving valuable outcomes. Research to date has not yet adequately examined the differences in the perceived difficulty of humans and AI. Thus, this work reviews recent research on the perceived difficulty in human-AI interaction and contributing factors to consistently compare each agent's perceived difficulty, e.g., creating the same prerequisites. Furthermore, we present an experimental design to thoroughly examine the perceived difficulty of both agents and contribute to a better understanding of the design of such systems.