# How Accurate Does It Feel? -- Human Perception of Different Types of Classification Mistakes

arXiv: https://arxiv.org/abs/2302.06413 · HTML: https://mlsanity.com/paper/2302.06413/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Feb 2023.

## Built on by (most cited first, 4 of 4)

- [One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generations](https://mlsanity.com/paper/2405.05581/) (2024)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [A Comparison of Human and Machine Learning Errors in Face Recognition](https://mlsanity.com/paper/2502.11337/) (2025)
- [Effects of Generative AI Errors on User Reliance Across Task Difficulty](https://mlsanity.com/paper/2604.04319/) (2026)

## Abstract (arXiv)

Supervised machine learning utilizes large datasets, often with ground truth labels annotated by humans. While some data points are easy to classify, others are hard to classify, which reduces the inter-annotator agreement. This causes noise for the classifier and might affect the user's perception of the classifier's performance. In our research, we investigated whether the classification difficulty of a data point influences how strongly a prediction mistake reduces the "perceived accuracy". In an experimental online study, 225 participants interacted with three fictive classifiers with equal accuracy (73%). The classifiers made prediction mistakes on three different types of data points (easy, difficult, impossible). After the interaction, participants judged the classifier's accuracy. We found that not all prediction mistakes reduced the perceived accuracy equally. Furthermore, the perceived accuracy differed significantly from the calculated accuracy. To conclude, accuracy and related measures seem unsuitable to represent how users perceive the performance of classifiers.