# Leveraging Annotator Disagreement for Text Classification

arXiv: https://arxiv.org/abs/2409.17577 · HTML: https://mlsanity.com/paper/2409.17577/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Identifying and Correcting Label Bias in Machine Learning](https://mlsanity.com/paper/1901.04966/) (2019)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [Who Said What: Modeling Individual Labelers Improves Classification](https://mlsanity.com/paper/1703.08774/) (2017)
- [Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information](https://mlsanity.com/paper/2301.05036/) (2023)
- [Detecting Stance in Media on Global Warming](https://mlsanity.com/paper/2010.15149/) (2020)
- [ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AI](https://mlsanity.com/paper/2109.09483/) (2021)
- [Can Unconditional Language Models Recover Arbitrary Sentences?](https://mlsanity.com/paper/1907.04944/) (2019)
- [Degree based Classification of Harmful Speech using Twitter Data](https://mlsanity.com/paper/1806.04197/) (2018)
- [Soft-prompt Tuning for Large Language Models to Evaluate Bias](https://mlsanity.com/paper/2306.04735/) (2023)

## Built on by (most cited first, 8 of 8)

- [Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluation](https://mlsanity.com/paper/2603.06865/) (2026)
- [A Collaborative Content Moderation Framework for Toxicity Detection based on Conformalized Estimates of Annotation Disagreement](https://mlsanity.com/paper/2411.04090/) (2024)
- [STABLEVAL: Disagreement-Aware and Stable Evaluation of AI Systems](https://mlsanity.com/paper/2605.02122/) (2026)
- [Fine-Grained Perspectives: Modeling Explanations with Annotator-Specific Rationales](https://mlsanity.com/paper/2604.21667/) (2026)
- [Understanding Annotator Safety Policy with Interpretability](https://mlsanity.com/paper/2605.05329/) (2026)
- [Beyond Majority Voting: Agreement-Based Clustering to Model Annotator Perspectives in Subjective NLP Tasks](https://mlsanity.com/paper/2605.09955/) (2026)
- [Evaluating Large Language Models for Antisemitic Incident Classification](https://mlsanity.com/paper/2607.04890/) (2026)
- [Ensemble Diversity Optimization for Subjective Supervision](https://mlsanity.com/paper/2607.08493/) (2026)

## Abstract (arXiv)

It is common practice in text classification to only use one majority label for model training even if a dataset has been annotated by multiple annotators. Doing so can remove valuable nuances and diverse perspectives inherent in the annotators' assessments. This paper proposes and compares three different strategies to leverage annotator disagreement for text classification: a probability-based multi-label method, an ensemble system, and instruction tuning. All three approaches are evaluated on the tasks of hate speech and abusive conversation detection, which inherently entail a high degree of subjectivity. Moreover, to evaluate the effectiveness of embracing annotation disagreements for model training, we conduct an online survey that compares the performance of the multi-label model against a baseline model, which is trained with the majority label. The results show that in hate speech detection, the multi-label method outperforms the other two approaches, while in abusive conversation detection, instruction tuning achieves the best performance. The results of the survey also show that the outputs from the multi-label models are considered a better representation of the texts than the single-label model.