# The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

arXiv: https://arxiv.org/abs/2405.05860 · HTML: https://mlsanity.com/paper/2405.05860/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 91 references that are in the map.
- First posted on arXiv 9 May 2024.

## Builds on (most cited first, 30 of 31)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasets](https://mlsanity.com/paper/1908.07898/) (2019)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [MultiVerS: Improving scientific claim verification with weak supervision and full-document context](https://mlsanity.com/paper/2112.01640/) (2021)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [A Survey on the Fairness of Recommender Systems](https://mlsanity.com/paper/2206.03761/) (2022)
- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Tasks](https://mlsanity.com/paper/2305.06626/) (2023)
- [The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice](https://mlsanity.com/paper/2310.00907/) (2023)
- [Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development](https://mlsanity.com/paper/2108.04308/) (2021)
- [Stop Measuring Calibration When Humans Disagree](https://mlsanity.com/paper/2210.16133/) (2022)
- [Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation](https://mlsanity.com/paper/2205.00501/) (2022)
- [Privacy-Preserving Machine Learning: Methods, Challenges and Directions](https://mlsanity.com/paper/2108.04417/) (2021)
- [Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Information](https://mlsanity.com/paper/2301.05036/) (2023)
- [The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographics](https://mlsanity.com/paper/2306.11559/) (2023)
- [You Are What You Annotate: Towards Better Models through Annotator Representations](https://mlsanity.com/paper/2305.14663/) (2023)
- [Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection](https://mlsanity.com/paper/2106.15896/) (2021)
- [Disembodied Machine Learning: On the Illusion of Objectivity in NLP](https://mlsanity.com/paper/2101.11974/) (2021)
- [Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoning](https://mlsanity.com/paper/2104.08676/) (2021)
- [Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreement](https://mlsanity.com/paper/2301.10684/) (2023)
- [Unifying Data Perspectivism and Personalization: An Application to Social Norms](https://mlsanity.com/paper/2210.14531/) (2022)
- [Incorporating Worker Perspectives into MTurk Annotation Practices for NLP](https://mlsanity.com/paper/2311.02802/) (2023)
- [A Crowdsourced Frame Disambiguation Corpus with Ambiguity](https://mlsanity.com/paper/1904.06101/) (2019)
- [Being Right for Whose Right Reasons?](https://mlsanity.com/paper/2306.00639/) (2023)
- [Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarization](https://mlsanity.com/paper/2212.10397/) (2022)
- [Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote method](https://mlsanity.com/paper/2305.20042/) (2023)
- [Principled Evaluation with Human Labels: One Rater at a Time and Rater Equivalence](https://mlsanity.com/paper/2106.01254/) (2021)

## Built on by (most cited first, 30 of 38)

- [Diverging Preferences: When do Annotators Disagree and do Models Know?](https://mlsanity.com/paper/2410.14632/) (2024)
- [Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptions](https://mlsanity.com/paper/2502.20897/) (2025)
- [Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation](https://mlsanity.com/paper/2603.25001/) (2026)
- [Which Demographics do LLMs Default to During Annotation?](https://mlsanity.com/paper/2410.08820/) (2024)
- [Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation](https://mlsanity.com/paper/2410.23448/) (2024)
- [Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensiveness](https://mlsanity.com/paper/2411.08977/) (2024)
- [NUTMEG: Separating Signal From Noise in Annotator Disagreement](https://mlsanity.com/paper/2507.18890/) (2025)
- [What is Stigma Attributed to? A Theory-Grounded, Expert-Annotated Interview Corpus for Demystifying Mental-Health Stigma](https://mlsanity.com/paper/2505.12727/) (2025)
- [Who's Laughing Now? An Overview of Computational Humour Generation and Explanation](https://mlsanity.com/paper/2509.21175/) (2025)
- [PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harm](https://mlsanity.com/paper/2601.08951/) (2026)
- [Sensitive Content Classification in Social Media: A Holistic Resource and Evaluation](https://mlsanity.com/paper/2411.19832/) (2024)
- [Attention to Non-Adopters](https://mlsanity.com/paper/2510.15951/) (2025)
- [LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gaps](https://mlsanity.com/paper/2604.27345/) (2026)
- [A Framework for Evaluating LLMs Under Task Indeterminacy](https://mlsanity.com/paper/2411.13760/) (2024)
- [Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection](https://mlsanity.com/paper/2412.11745/) (2024)
- [The Impact of Annotator Personas on LLM Behavior Across the Perspectivism Spectrum](https://mlsanity.com/paper/2508.17164/) (2025)
- [Learning to Generate and Evaluate Fact-checking Explanations with Transformers](https://mlsanity.com/paper/2410.15669/) (2024)
- [ORCA: Open-ended Response Correctness Assessment for Audio Question Answering](https://mlsanity.com/paper/2512.09066/) (2025)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)
- [Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfer](https://mlsanity.com/paper/2606.11018/) (2026)
- [Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replication](https://mlsanity.com/paper/2501.06826/) (2025)
- [ModelCitizens: Representing Community Voices in Online Safety](https://mlsanity.com/paper/2507.05455/) (2025)
- [How Annotation Trains Annotators: Competence Development in Social Influence Recognition](https://mlsanity.com/paper/2604.02951/) (2026)
- [Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generation](https://mlsanity.com/paper/2606.07936/) (2026)
- [Eigenism: Ethics for a Human-AI Future](https://mlsanity.com/paper/2606.12420/) (2026)
- [Splits! Flexible Sociocultural Linguistic Investigation at Scale](https://mlsanity.com/paper/2504.04640/) (2025)
- [From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Models](https://mlsanity.com/paper/2507.17922/) (2025)
- [Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributions](https://mlsanity.com/paper/2509.08217/) (2025)
- [Hypernetworks for Perspectivist Adaptation](https://mlsanity.com/paper/2510.13259/) (2025)
- [Examining the Utility of Self-disclosure Types for Modeling Annotators of Social Norms](https://mlsanity.com/paper/2512.16034/) (2025)

## Abstract (arXiv)

Longstanding data labeling practices in machine learning involve collecting and aggregating labels from multiple annotators. But what should we do when annotators disagree? Though annotator disagreement has long been seen as a problem to minimize, new perspectivist approaches challenge this assumption by treating disagreement as a valuable source of information. In this position paper, we examine practices and assumptions surrounding the causes of disagreement--some challenged by perspectivist approaches, and some that remain to be addressed--as well as practical and normative challenges for work operating under these assumptions. We conclude with recommendations for the data labeling pipeline and avenues for future research engaging with subjectivity and disagreement.