mlsanityData to 7 Oct 2026

Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection

arXiv 2111.07997 ↗First posted 15 Nov 2021cs.CL cs.HCcited by 258

Not on the map. Why?

Abstract (arXiv)

The perceived toxicity of language can vary based on someone's identity and beliefs, but this variation is often ignored when collecting toxic language datasets, resulting in dataset and model biases. We seek to understand the who, why, and what behind biases in toxicity annotations. In two online studies with demographically and politically diverse participants, we investigate the effect of annotator identities (who) and beliefs (why), drawing from social psychology research about hate speech, free speech, racist beliefs, political leaning, and more. We disentangle what is annotated as toxic by considering posts with three characteristics: anti-Black language, African American English (AAE) dialect, and vulgarity. Our results show strong associations between annotator identity and beliefs and their ratings of toxicity. Notably, more conservative annotators and those who scored highly on our scale for racist beliefs were less likely to rate anti-Black language as toxic, but more likely to rate AAE as toxic. We additionally present a case study illustrating how a popular toxicity detection system's ratings inherently reflect only specific beliefs and perspectives. Our findings call for contextualizing toxicity labels in social variables, which raises immense implications for toxic language annotation and detection.

Key facts. Cited by 258 papers in the map (data through 7 Oct 2026). Builds on 21 of its 116 references that are in the map. First posted on arXiv 15 Nov 2021.

Builds on

21 of 116 references.

References · 21

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  3. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  4. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  5. 2019Predicting the Type and Target of Offensive Posts in Social Mediacited by 268
  6. 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 265
  7. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 217
  8. 2016Ex Machina: Personal Attacks Seen at Scalecited by 208
  9. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 208
  10. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  11. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  12. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
9 more
  1. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 123
  2. 2017Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisiscited by 95
  3. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 95
  4. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  5. 2016Analyzing the Targets of Hate in Online Social Mediacited by 48
  6. 2021Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detectioncited by 38
  7. 2021Disembodied Machine Learning: On the Illusion of Objectivity in NLPcited by 37
  8. 2018Data Science as Political Action: Grounding Data Science in a Politics of Justicecited by 14
  9. 2021Hate Speech Classifiers Learn Human-Like Social Stereotypescited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 28; 2023: 66; 2024: 52; 2025: 54; 2026: 57.

258 papers.

Sort:

By area:

  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2023Whose Opinions Do Language Models Reflect?cited by 759
  4. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  5. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  6. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 257
  7. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  8. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  9. 2024Quantifying the Persona Effect in LLM Simulationscited by 164
  10. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  11. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  12. 2023Auditing large language models: a three-layered approachcited by 135
38 more
  1. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  2. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  3. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  4. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  5. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
  6. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  7. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  8. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  9. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  10. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  11. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  12. 2022Mix and Match: Learning-free Controllable Text Generation using Energy Language Modelscited by 66
  13. 2024Survey of Bias In Text-to-Image Generation: Definition, Evaluation, and Mitigationcited by 66
  14. 2022POTATO: The Portable Text Annotation Toolcited by 64
  15. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 59
  16. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  17. 2022Describing Differences between Text Distributions with Natural Languagecited by 54
  18. 2022Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selectioncited by 52
  19. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  20. 2022Multilingual HateCheck: Functional Tests for Multilingual Hate Speech Detection Modelscited by 51
  21. 2023When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Datasetcited by 51
  22. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  23. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  24. 2023"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social mediacited by 45
  25. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 45
  26. 2023Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysiscited by 44
  27. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  28. 2022Handling and Presenting Harmful Text in NLP Researchcited by 43
  29. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  30. 2022Towards Identifying Social Bias in Dialog Systems: Frame, Datasets, and Benchmarkscited by 36
  31. 2022Detoxifying Text with MaRCo: Controllable Revision with Experts and Anti-Expertscited by 36
  32. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36
  33. 2023Rethinking Machine Ethics -- Can LLMs Perform Moral Reasoning through the Lens of Moral Theories?cited by 36
  34. 2026Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluationscited by 36
  35. 2023COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statementscited by 34
  36. 2022Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimizationcited by 32
  37. 2024Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targetscited by 31
  38. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 31

50 most cited of 258 All 258 (JSON)