# Quantifying the Impact of Cognitive Biases in Question-Answering Systems

arXiv: https://arxiv.org/abs/1909.09633 · HTML: https://mlsanity.com/paper/1909.09633/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Sep 2019.

## Built on by (most cited first, 3 of 3)

- [ImageInWords: Unlocking Hyper-Detailed Image Descriptions](https://mlsanity.com/paper/2405.02793/) (2024)
- [NoisywikiHow: A Benchmark for Learning with Real-world Noisy Labels in Natural Language Processing](https://mlsanity.com/paper/2305.10709/) (2023)
- [Posts of Peril: Detecting Information About Hazards in Text](https://mlsanity.com/paper/2405.17838/) (2024)

## Abstract (arXiv)

Crowdsourcing can identify high-quality solutions to problems; however, individual decisions are constrained by cognitive biases. We investigate some of these biases in an experimental model of a question-answering system. In both natural and controlled experiments, we observe a strong position bias in favor of answers appearing earlier in a list of choices. This effect is enhanced by three cognitive factors: the attention an answer receives, its perceived popularity, and cognitive load, measured by the number of choices a user has to process. While separately weak, these effects synergistically amplify position bias and decouple user choices of best answers from their intrinsic quality. We end our paper by discussing the novel ways we can apply these findings to substantially improve how high-quality answers are found in question-answering systems.