# Majority Vote Silences Minority Values: Annotator Disagreement at the Hate/Offensive Boundary in HateXplain

arXiv: https://arxiv.org/abs/2606.28772 · HTML: https://mlsanity.com/paper/2606.28772/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 11 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Builds on (most cited first, 7 of 7)

- [HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection](https://mlsanity.com/paper/2012.10289/) (2020)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [Jury Learning: Integrating Dissenting Voices into Machine Learning Models](https://mlsanity.com/paper/2202.02950/) (2022)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [DICES Dataset: Diversity in Conversational AI Evaluation for Safety](https://mlsanity.com/paper/2306.11247/) (2023)

## Abstract (arXiv)

Hate speech annotation pipelines routinely collapse annotator disagreement into majority vote labels before training. We show that this aggregation is not neutral: 42.6% of all annotator disagreement in HateXplain concentrates specifically at the hate/offensive boundary, a pattern consistent with annotators applying different thresholds for where hate begins (chi-squared = 135.199, df = 2, p < 0.0001). Both a hard-label BERT model (Model A) and a soft-label model (Model B) drop 22 percentage points in accuracy from agreed posts (~80%) to disagreement posts (~58%), confirmed at p < 0.0001. A per-annotator multi-head model (Model C) widens this gap further to 28 points while collapsing offensive disagreement accuracy to 0.245. Critically, Model A expresses significantly higher confidence on boundary case errors than Model C (0.710 vs. 0.495, p < 0.0001), meaning standard evaluation metrics will not detect the failure. Three downstream interventions of increasing sophistication all fail to recover boundary accuracy. We argue the problem is structural. Majority vote presents a contested judgment as ground truth, and models inherit that false certainty. The intervention must be upstream in annotation design.