# The Origin and Value of Disagreement Among Data Labelers: A Case Study of the Individual Difference in Hate Speech Annotation

arXiv: https://arxiv.org/abs/2112.04030 · HTML: https://mlsanity.com/paper/2112.04030/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Dec 2021.

## Built on by (most cited first, 15 of 15)

- [Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets](https://mlsanity.com/paper/2410.07991/) (2024)
- [Beyond Consensus: Perspectivist Modeling and Evaluation of Annotator Disagreement in NLP](https://mlsanity.com/paper/2601.09065/) (2026)
- [PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization](https://mlsanity.com/paper/2407.18078/) (2024)
- [Subjective Crowd Disagreements for Subjective Data: Uncovering Meaningful CrowdOpinion with Population-level Learning](https://mlsanity.com/paper/2307.10189/) (2023)
- [Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensiveness](https://mlsanity.com/paper/2411.08977/) (2024)
- [BAN-PL: a Novel Polish Dataset of Banned Harmful and Offensive Content from Wykop.pl web service](https://mlsanity.com/paper/2308.10592/) (2023)
- [A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicity](https://mlsanity.com/paper/2311.04345/) (2023)
- [Cost-Efficient Subjective Task Annotation and Modeling through Few-Shot Annotator Adaptation](https://mlsanity.com/paper/2402.14101/) (2024)
- [Finding Pareto Trade-offs in Fair and Accurate Detection of Toxic Speech](https://mlsanity.com/paper/2204.07661/) (2022)
- [Dealing with Annotator Disagreement in Hate Speech Classification](https://mlsanity.com/paper/2502.08266/) (2025)
- [Modeling Human Perspectives with Socio-Demographic Representations](https://mlsanity.com/paper/2604.18069/) (2026)
- [Quantifying and Predicting Disagreement in Graded Human Ratings](https://mlsanity.com/paper/2605.01168/) (2026)
- [Examining the Utility of Self-disclosure Types for Modeling Annotators of Social Norms](https://mlsanity.com/paper/2512.16034/) (2025)
- [Improving Implicit Hate Speech Detection via a Community-Driven Multi-Agent Framework](https://mlsanity.com/paper/2601.09342/) (2026)
- [Human Label Variation in Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2602.22723/) (2026)

## Abstract (arXiv)

Human annotated data is the cornerstone of today's artificial intelligence efforts, yet data labeling processes can be complicated and expensive, especially when human labelers disagree with each other. The current work practice is to use majority-voted labels to overrule the disagreement. However, in the subjective data labeling tasks such as hate speech annotation, disagreement among individual labelers can be difficult to resolve. In this paper, we explored why such disagreements occur using a mixed-method approach - including interviews with experts, concept mapping exercises, and self-reporting items - to develop a multidimensional scale for distilling the process of how annotators label a hate speech corpus. We tested this scale with 170 annotators in a hate speech annotation task. Results showed that our scale can reveal facets of individual differences among annotators (e.g., age, personality, etc.), and these facets' relationships to an annotator's final label decision of an instance. We suggest that this work contributes to the understanding of how humans annotate data. The proposed scale can potentially improve the value of the currently discarded minority-vote labels.