# Studying the Effects of Cognitive Biases in Evaluation of Conversational Agents

arXiv: https://arxiv.org/abs/2002.07927 · HTML: https://mlsanity.com/paper/2002.07927/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Feb 2020.

## Built on by (most cited first, 8 of 8)

- [Achieving Reliable Human Assessment of Open-Domain Dialogue Systems](https://mlsanity.com/paper/2203.05899/) (2022)
- [Towards Standard Criteria for human evaluation of Chatbots: A Survey](https://mlsanity.com/paper/2105.11197/) (2021)
- [Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection](https://mlsanity.com/paper/2308.12885/) (2023)
- [EmPO: Emotion Grounding for Empathetic Response Generation through Preference Optimization](https://mlsanity.com/paper/2406.19071/) (2024)
- [Trapped by Expectations: Functional Fixedness in LLM-Enabled Chat Search](https://mlsanity.com/paper/2504.02074/) (2025)
- [Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs](https://mlsanity.com/paper/2404.12994/) (2024)
- [Evaluating and Improving Value Judgments in AI: A Scenario-Based Study on Large Language Models' Depiction of Social Conventions](https://mlsanity.com/paper/2311.09230/) (2023)
- [Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2404.09980/) (2024)

## Abstract (arXiv)

Humans quite frequently interact with conversational agents. The rapid advancement in generative language modeling through neural networks has helped advance the creation of intelligent conversational agents. Researchers typically evaluate the output of their models through crowdsourced judgments, but there are no established best practices for conducting such studies. Moreover, it is unclear if cognitive biases in decision-making are affecting crowdsourced workers' judgments when they undertake these tasks. To investigate, we conducted a between-subjects study with 77 crowdsourced workers to understand the role of cognitive biases, specifically anchoring bias, when humans are asked to evaluate the output of conversational agents. Our results provide insight into how best to evaluate conversational agents. We find increased consistency in ratings across two experimental conditions may be a result of anchoring bias. We also determine that external factors such as time and prior experience in similar tasks have effects on inter-rater consistency.