mlsanityData to 7 Oct 2026

Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain

arXiv 2606.28772 ↗First posted 27 Jun 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training. We show that this aggregation is not neutral: 42.6% of all annotator disagreement in HateXplain concentrates specifically at the hate/offensive boundary, a pattern consistent with annotators applying different thresholds for where hate begins (chi-squared = 135.199, df = 2, p < 0.0001). Both a hard-label BERT model (Model A) and a soft-label model (Model B) drop 22 percentage points in accuracy from agreed posts (~80%) to disagreement posts (~58%), confirmed at p < 0.0001. A per-annotator multi-head model (Model C) widens this gap further to 28 points while collapsing offensive disagreement accuracy to 0.245. Critically, Model A expresses significantly higher confidence on boundary case errors than Model C (0.710 vs. 0.495, p < 0.0001), meaning standard evaluation metrics will not detect the failure. Three downstream interventions of increasing sophistication all fail to recover boundary accuracy. We argue the problem is structural. Majority vote presents a contested judgment as ground truth, and models inherit that false certainty. The intervention must be upstream in annotation design.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 11 references that are in the map. First posted on arXiv 27 Jun 2026.

Builds on

7 of 11 references.

References · 7

  1. 2020HateXplain: A Benchmark Dataset for Explainable Hate Speech Detectioncited by 403
  2. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 306
  3. 2024A Roadmap to Pluralistic Alignmentcited by 175
  4. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  5. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
  6. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  7. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86

Built on by

Not cited yet.