mlsanityData to 14 Oct 2026

Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations

arXiv 2110.05719 ↗First posted 12 Oct 2021cs.CL cs.CYcited by 314

Not on the map. Why?

Cited by 22 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Majority voting and averaging are common approaches employed to resolve annotator disagreements and derive single ground truth labels from multiple annotations. However, annotators may systematically disagree with one another, often reflecting their individual biases and values, especially in the case of subjective tasks such as detecting affect, aggression, and hate speech. Annotator disagreements may capture important nuances in such tasks that are often ignored while aggregating annotations to a single ground truth. In order to address this, we investigate the efficacy of multi-annotator models. In particular, our multi-task based approach treats predicting each annotators' judgements as separate subtasks, while sharing a common learned representation of the task. We show that this approach yields same or better performance than aggregating labels in the data prior to training across seven different binary classification tasks. Our approach also provides a way to estimate uncertainty in predictions, which we demonstrate better correlate with annotation disagreements than traditional methods. Being able to model uncertainty is especially useful in deployment scenarios where knowing when not to make a prediction is important.

Key facts. Cited by 314 papers in the map (data through 14 Oct 2026). Builds on 24 of its 96 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

24 of 96 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  5. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  6. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  7. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  8. 2020Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detectioncited by 221
  9. 2019Racial Bias in Hate Speech and Abusive Language Detection Datasetscited by 209
  10. 2020Social Biases in NLP Models as Barriers for Persons with Disabilitiescited by 182
  11. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  12. 2019Emotion Recognition in Conversation: Research Challenges, Datasets, and Recent Advancescited by 119
12 more
  1. 2017Understanding Abuse: A Typology of Abusive Language Detection Subtaskscited by 116
  2. 2019CXPlain: Causal Explanations for Model Interpretation under Uncertaintycited by 98
  3. 2019A BERT-Based Transfer Learning Approach for Hate Speech Detection in Online Social Mediacited by 96
  4. 2021Challenges in Automated Debiasing for Toxic Language Detectioncited by 96
  5. 2019Perturbation Sensitivity Analysis to Detect Unintended Model Biasescited by 74
  6. 2019A Just and Comprehensive Strategy for Using NLP to Address Online Abusecited by 49
  7. 2020Detecting Stance in Media on Global Warmingcited by 48
  8. 2019Tackling Online Abuse: A Survey of Automated Abuse Detection Methodscited by 33
  9. 2020Six Attributes of Unhealthy Conversationcited by 19
  10. 2018Crowd disagreement about medical images is informativecited by 13
  11. 2019Characterizing Sources of Uncertainty to Proxy Calibration and Disambiguate Annotator and Data Biascited by 11
  12. 2015Statistical modality tagging from rule-based annotations and crowdsourcingcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 29; 2023: 65; 2024: 72; 2025: 55; 2026: 89.

314 papers.

Sort:

By area:

  1. 2023Whose Opinions Do Language Models Reflect?cited by 773
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  3. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 437
  4. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  5. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  6. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  7. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  8. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  9. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 155
  10. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  11. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
  12. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
38 more
  1. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  2. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  3. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
  4. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 90
  5. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  6. 2023AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languagescited by 68
  7. 2022NaijaSenti: A Nigerian Twitter Sentiment Corpus for Multilingual Sentiment Analysiscited by 64
  8. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  9. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 54
  10. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  11. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 52
  12. 2023SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)cited by 52
  13. 2022Evaluation Gaps in Machine Learning Practicecited by 49
  14. 2024Beyond Preferences in AI Alignmentcited by 49
  15. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  16. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 46
  17. 2023Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysiscited by 45
  18. 2023MoCa: Measuring Human-Language Model Alignment on Causal and Moral Judgment Taskscited by 44
  19. 2022Is one annotation enough? A data-centric image classification benchmark for noisy and ambiguous label estimationcited by 43
  20. 2024VariErr NLI: Separating Annotation Error from Human Label Variationcited by 42
  21. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  22. 2024KorNAT: LLM Alignment Benchmark for Korean Social Values and Common Knowledgecited by 33
  23. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 33
  24. 2022Does Moral Code Have a Moral Code? Probing Delphi's Moral Philosophycited by 31
  25. 2022Investigating Reasons for Disagreement in Natural Language Inferencecited by 29
  26. 2023Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Taskscited by 29
  27. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  28. 2024Emotion Analysis in NLP: Trends, Gaps and Roadmap for Future Directionscited by 27
  29. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 26
  30. 2023Disentangling Perceptions of Offensiveness: Cultural and Moral Correlatescited by 26
  31. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  32. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
  33. 2024SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypescited by 24
  34. 2024Extrinsic Evaluation of Cultural Competence in Large Language Modelscited by 24
  35. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 23
  36. 2024Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personascited by 23
  37. 2024NADI 2024: The Fifth Nuanced Arabic Dialect Identification Shared Taskcited by 23
  38. 2022Unifying Data Perspectivism and Personalization: An Application to Social Normscited by 22

50 most cited of 314 All 314 (JSON)