# Bias in the Loop: How Humans Evaluate AI-Generated Suggestions

arXiv: https://arxiv.org/abs/2509.08514 · HTML: https://mlsanity.com/paper/2509.08514/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 75 references that are in the map.
- First posted on arXiv 10 Sep 2025.

## Builds on (most cited first, 20 of 20)

- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search](https://mlsanity.com/paper/1905.01989/) (2019)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making](https://mlsanity.com/paper/2010.07938/) (2020)
- [LayoutCopilot: An LLM-powered Multi-agent Collaborative Framework for Interactive Analog Layout Design](https://mlsanity.com/paper/2406.18873/) (2024)
- [A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregation](https://mlsanity.com/paper/2401.09760/) (2024)
- [D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias](https://mlsanity.com/paper/2208.05126/) (2022)
- [Empirica: a virtual lab for high-throughput macro-level experiments](https://mlsanity.com/paper/2006.11398/) (2020)
- [GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction](https://mlsanity.com/paper/2405.15760/) (2024)
- [Annotation Sensitivity: Training Data Collection Methods Affect Model Performance](https://mlsanity.com/paper/2311.14212/) (2023)
- [Position: Insights from Survey Methodology can Improve Training Data](https://mlsanity.com/paper/2403.01208/) (2024)
- [Incentivizing High Quality Crowdwork](https://mlsanity.com/paper/1503.05897/) (2015)
- [Quality and Efficiency of Manual Annotation: Pre-annotation Bias](https://mlsanity.com/paper/2306.09307/) (2023)
- [Don't be Fooled: The Misinformation Effect of Explanations in Human-AI Collaboration](https://mlsanity.com/paper/2409.12809/) (2024)
- [Bending the Automation Bias Curve: A Study of Human and AI-based Decision Making in National Security Contexts](https://mlsanity.com/paper/2306.16507/) (2023)
- [Investigating Annotator Bias in Large Language Models for Hate Speech Detection](https://mlsanity.com/paper/2406.11109/) (2024)
- [Leveraging Large-scale Multimedia Datasets to Refine Content Moderation Models](https://mlsanity.com/paper/2212.00668/) (2022)
- [Attention-Aware Answers of the Crowd](https://mlsanity.com/paper/1912.11238/) (2019)
- [BiasEye: A Bias-Aware Real-time Interactive Material Screening System for Impartial Candidate Assessment](https://mlsanity.com/paper/2402.09148/) (2024)

## Built on by (most cited first, 4 of 4)

- [Critical or Compliant? The Double-Edged Sword of Reasoning in Chain-of-Thought Explanations](https://mlsanity.com/paper/2511.12001/) (2025)
- [The Productivity-Reliability Paradox: Specification-Driven Governance for AI-Augmented Software Development](https://mlsanity.com/paper/2605.01160/) (2026)
- [Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation](https://mlsanity.com/paper/2510.13884/) (2025)
- [ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation](https://mlsanity.com/paper/2603.21094/) (2026)

## Abstract (arXiv)

Human-AI collaboration increasingly drives decision-making across industries, from medical diagnosis to content moderation. While AI systems promise efficiency gains by providing automated suggestions for human review, these workflows can trigger cognitive biases that degrade performance. We know little about the psychological factors that determine when these collaborations succeed or fail. We conducted a randomized experiment with 2,784 participants to examine how task design and individual characteristics shape human responses to AI-generated suggestions. Using a controlled annotation task, we manipulated three factors: AI suggestion quality in the first three instances, task burden through required corrections, and performance-based financial incentives. We collected demographics, attitudes toward AI, and behavioral data to assess four performance metrics: accuracy, correction activity, overcorrection, and undercorrection. Two patterns emerged that challenge conventional assumptions about human-AI collaboration. First, requiring corrections for flagged AI errors reduced engagement and increased the tendency to accept incorrect suggestions, demonstrating how cognitive shortcuts influence collaborative outcomes. Second, individual attitudes toward AI emerged as the strongest predictor of performance, surpassing demographic factors. Participants skeptical of AI detected errors more reliably and achieved higher accuracy, while those favorable toward automation exhibited dangerous overreliance on algorithmic suggestions. The findings reveal that successful human-AI collaboration depends not only on algorithmic performance but also on who reviews AI outputs and how review processes are structured. Effective human-AI collaborations require consideration of human psychology: selecting diverse evaluator samples, measuring attitudes, and designing workflows that counteract cognitive biases.