mlsanityData to 14 Oct 2026

Blind Spots and Biases: Exploring the Role of Annotator Cognitive Biases in NLP

arXiv 2404.19071 ↗First posted 29 Apr 2024cs.HC cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

With the rapid proliferation of artificial intelligence, there is growing concern over its potential to exacerbate existing biases and societal disparities and introduce novel ones. This issue has prompted widespread attention from academia, policymakers, industry, and civil society. While evidence suggests that integrating human perspectives can mitigate bias-related issues in AI systems, it also introduces challenges associated with cognitive biases inherent in human decision-making. Our research focuses on reviewing existing methodologies and ongoing investigations aimed at understanding annotation attributes that contribute to bias.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 22 of its 62 references that are in the map. First posted on arXiv 29 Apr 2024.

Builds on

22 of 62 references.

References · 22

  1. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  2. 2019A Survey on Bias and Fairness in Machine Learningcited by 1,917
  3. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  5. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  6. 2014Certifying and removing disparate impactcited by 715
  7. 2018A Reductions Approach to Fair Classificationcited by 587
  8. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 586
  9. 2017On Fairness and Calibrationcited by 406
  10. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  11. 2016Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatmentcited by 369
  12. 2019Mitigating Gender Bias in Natural Language Processing: Literature Reviewcited by 287
10 more
  1. 2021A Survey of Human-in-the-loop for Machine Learningcited by 215
  2. 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 193
  3. 2016Fairness in Learning: Classic and Contextual Banditscited by 168
  4. 2021Re-imagining Algorithmic Fairness in India and Beyondcited by 104
  5. 2022CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotationcited by 59
  6. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  7. 2017Calibrated Fairness in Banditscited by 43
  8. 2023ChatGPT Needs SPADE (Sustainability, PrivAcy, Digital divide, and Ethics) Evaluation: A Reviewcited by 28
  9. 2023Out of Context: Investigating the Bias and Fairness Concerns of "Artificial Intelligence as a Service"cited by 15
  10. 2023"I'm" Lost in Translation: Pronoun Missteps in Crowdsourced Data Setscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 3.

11 papers.

  1. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  2. 2024A Comprehensive Survey of Bias in LLMs: Current Landscape and Future Directionscited by 27
  3. 2025Addressing Bias in Generative AI: Challenges and Research Opportunities in Information Managementcited by 7
  4. 2025BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Contextcited by 6
  5. 2026Can Humans Dream of Electric Sheep? Human-Written Samples for Fine-Grained Vision-and-Language Hallucination Benchmarkingcited by 4
  6. 2025Deceptive Humor: A Synthetic Multilingual Benchmark Dataset for Bridging Fabricated Claims with Humorous Contentcited by 3
  7. 2024Uncovering Biases with Reflective Large Language Modelscited by 2
  8. 2024LLM-Human Pipeline for Cultural Context Grounding of Conversationscited by 2
  9. 2026Expert Evaluation and the Limits of Human Feedback in Mental Health AI Safety Testingcited by 2
  10. 2025Keeping it Local, Tiny and Real: Automated Report Generation on Edge Computing Devices for Mechatronic-Based Cognitive Systemscited by 1
  11. 2026Scalable Evaluation of the Realism of Synthetic Environmental Augmentations in Imagescited by 0