Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain
Not on the map. Why?
Abstract (arXiv)
Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training. We show that this aggregation is not neutral: 42.6% of all annotator disagreement in HateXplain concentrates specifically at the hate/offensive boundary, a pattern consistent with annotators applying different thresholds for where hate begins (chi-squared = 135.199, df = 2, p < 0.0001). Both a hard-label BERT model (Model A) and a soft-label model (Model B) drop 22 percentage points in accuracy from agreed posts (~80%) to disagreement posts (~58%), confirmed at p < 0.0001. A per-annotator multi-head model (Model C) widens this gap further to 28 points while collapsing offensive disagreement accuracy to 0.245. Critically, Model A expresses significantly higher confidence on boundary case errors than Model C (0.710 vs. 0.495, p < 0.0001), meaning standard evaluation metrics will not detect the failure. Three downstream interventions of increasing sophistication all fail to recover boundary accuracy. We argue the problem is structural. Majority vote presents a contested judgment as ground truth, and models inherit that false certainty. The intervention must be upstream in annotation design.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 11 references that are in the map. First posted on arXiv 27 Jun 2026.
Builds on
7 of 11 references.
References · 7
- 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
- 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
- 2024A Roadmap to Pluralistic Alignmentcited by 175
- 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
- 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
- 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
- 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
Built on by
Not cited yet.