# Training and Evaluating with Human Label Variation: An Empirical Study

arXiv: https://arxiv.org/abs/2502.01891 · HTML: https://mlsanity.com/paper/2502.01891/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 50 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Builds on (most cited first, 14 of 14)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Human uncertainty makes classification more robust](https://mlsanity.com/paper/1908.07086/) (2019)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreement](https://mlsanity.com/paper/2109.13563/) (2021)
- [SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)](https://mlsanity.com/paper/2304.14803/) (2023)
- [TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations at Twitter](https://mlsanity.com/paper/2209.07562/) (2022)
- [You Are What You Annotate: Towards Better Models through Annotator Representations](https://mlsanity.com/paper/2305.14663/) (2023)
- [Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection](https://mlsanity.com/paper/2106.15896/) (2021)
- [Can Large Language Models Capture Dissenting Human Voices?](https://mlsanity.com/paper/2305.13788/) (2023)
- ["Seeing the Big through the Small": Can LLMs Approximate Human Judgment Distributions on NLI from a Few Explanations?](https://mlsanity.com/paper/2406.17600/) (2024)
- [iLab at SemEval-2023 Task 11 Le-Wi-Di: Modelling Disagreement or Modelling Perspectives?](https://mlsanity.com/paper/2305.06074/) (2023)
- [Evaluating Classification Systems Against Soft Labels with Fuzzy Precision and Recall](https://mlsanity.com/paper/2309.13938/) (2023)
- [To Aggregate or Not to Aggregate. That is the Question: A Case Study on Annotation Subjectivity in Span Prediction](https://mlsanity.com/paper/2408.02257/) (2024)

## Built on by (most cited first, 8 of 8)

- [Decoupling the Effect of Chain-of-Thought Reasoning: A Human Label Variation Perspective](https://mlsanity.com/paper/2601.03154/) (2026)
- [Threading the Needle: Reweaving Chain-of-Thought Reasoning to Explain Human Label Variation](https://mlsanity.com/paper/2505.23368/) (2025)
- [Disagreeing Rationales: Rethinking Classification and Explainability Evaluation in Hate Speech Detection](https://mlsanity.com/paper/2605.31563/) (2026)
- [Learning to vary: Teaching LMs to reproduce human linguistic variability in next-word prediction](https://mlsanity.com/paper/2509.17794/) (2025)
- [On the Interplay between Human Label Variation and Model Fairness](https://mlsanity.com/paper/2510.12036/) (2025)
- [Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP](https://mlsanity.com/paper/2603.28351/) (2026)
- [Instance-Level Costs for Nuanced Classifier Evaluation](https://mlsanity.com/paper/2605.03135/) (2026)
- [Metric-Dependent Annotation Saturation for Learning from Label Distributions](https://mlsanity.com/paper/2605.29797/) (2026)

## Abstract (arXiv)

Human label variation (HLV) challenges the standard assumption that a labelled instance has a single ground truth, instead embracing the natural variation in human annotation to train and evaluate models. While various training methods and metrics for HLV have been proposed, it is still unclear which methods and metrics perform best in what settings. We propose new evaluation metrics for HLV leveraging fuzzy set theory. Since these new proposed metrics are differentiable, we then in turn experiment with employing these metrics as training objectives. We conduct an extensive study over 6 HLV datasets testing 14 training methods and 6 evaluation metrics. We find that training on either disaggregated annotations or soft labels performs best across metrics, outperforming training using the proposed training objectives with differentiable metrics. We also show that our proposed soft micro F1 score is one of the best metrics for HLV data.