mlsanityData to 14 Oct 2026

The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels

arXiv 2405.05860 ↗First posted 9 May 2024cs.LG cs.CL cs.CYcited by 38

Not on the map. Why?

Abstract (arXiv)

Longstanding data labeling practices in machine learning involve collecting and aggregating labels from multiple annotators. But what should we do when annotators disagree? Though annotator disagreement has long been seen as a problem to minimize, new perspectivist approaches challenge this assumption by treating disagreement as a valuable source of information. In this position paper, we examine practices and assumptions surrounding the causes of disagreement--some challenged by perspectivist approaches, and some that remain to be addressed--as well as practical and normative challenges for work operating under these assumptions. We conclude with recommendations for the data labeling pipeline and avenues for future research engaging with subjectivity and disagreement.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 31 of its 91 references that are in the map. First posted on arXiv 9 May 2024.

Builds on

31 of 91 references.

References · 31

  1. 2018Datasheets for Datasetscited by 1,386
  2. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  3. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  4. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 175
  5. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  6. 2021MultiVerS: Improving scientific claim verification with weak supervision and full-document contextcited by 143
  7. 2021Toward a Perspectivist Turn in Ground Truthing for Predictive Computingcited by 130
  8. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
  9. 2022A Survey on the Fairness of Recommender Systemscited by 102
  10. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 94
  11. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 90
  12. 2023The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practicecited by 87
19 more
  1. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  2. 2022Stop Measuring Calibration When Humans Disagreecited by 71
  3. 2022Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotationcited by 69
  4. 2021Privacy-Preserving Machine Learning: Methods, Challenges and Directionscited by 58
  5. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  6. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 46
  7. 2023You Are What You Annotate: Towards Better Models through Annotator Representationscited by 40
  8. 2021Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detectioncited by 38
  9. 2021Disembodied Machine Learning: On the Illusion of Objectivity in NLPcited by 37
  10. 2021Distributed NLI: Learning to Predict Human Opinion Distributions for Language Reasoningcited by 31
  11. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 23
  12. 2022Unifying Data Perspectivism and Personalization: An Application to Social Normscited by 22
  13. 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 18
  14. 2019A Crowdsourced Frame Disambiguation Corpus with Ambiguitycited by 16
  15. 2023Being Right for Whose Right Reasons?cited by 12
  16. 2022Needle in a Haystack: An Analysis of High-Agreement Workers on MTurk for Summarizationcited by 10
  17. 2023Crowdsourcing subjective annotations using pairwise comparisons reduces bias and error compared to the majority-vote methodcited by 10
  18. 2021Principled Evaluation with Human Labels: One Rater at a Time and Rater Equivalencecited by 7
  19. 2023"Is a picture of a bird a bird": Policy recommendations for dealing with ambiguity in machine vision modelscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 14; 2026: 16.

38 papers.

Sort:

By area:

  1. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  2. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 33
  3. 2026Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluationcited by 24
  4. 2024Which Demographics do LLMs Default to During Annotation?cited by 10
  5. 2024Venire: A Machine Learning-Guided Panel Review System for Community Content Moderationcited by 7
  6. 2024Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensivenesscited by 7
  7. 2025NUTMEG: Separating Signal From Noise in Annotator Disagreementcited by 7
  8. 2025What is Stigma Attributed to? A Theory-Grounded, Expert-Annotated Interview Corpus for Demystifying Mental-Health Stigmacited by 6
  9. 2025Who's Laughing Now? An Overview of Computational Humour Generation and Explanationcited by 5
  10. 2026PluriHarms: Benchmarking the Full Spectrum of Human Judgments on AI Harmcited by 5
  11. 2024Sensitive Content Classification in Social Media: A Holistic Resource and Evaluationcited by 4
  12. 2025Attention to Non-Adopterscited by 4
26 more
  1. 2026LLMs Capture Emotion Labels, Not Emotion Uncertainty: Distributional Analysis and Calibration of Human-LLM Judgment Gapscited by 4
  2. 2024A Framework for Evaluating LLMs Under Task Indeterminacycited by 3
  3. 2024Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detectioncited by 3
  4. 2025The Impact of Annotator Personas on LLM Behavior Across the Perspectivism Spectrumcited by 3
  5. 2024Learning to Generate and Evaluate Fact-checking Explanations with Transformerscited by 2
  6. 2025ORCA: Open-ended Response Correctness Assessment for Audio Question Answeringcited by 2
  7. 2026From Ground Truth to Measurement: A Statistical Framework for Human Labelingcited by 2
  8. 2026Measuring Human Value Expression in Social Media Texts: Calibrated LLM Annotation and Encoder Transfercited by 2
  9. 2025Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replicationcited by 1
  10. 2025ModelCitizens: Representing Community Voices in Online Safetycited by 1
  11. 2026How Annotation Trains Annotators: Competence Development in Social Influence Recognitioncited by 1
  12. 2026Illusions of the Gold Standard: A Large-scale Analysis of Human Evaluation Protocols for Long-form Text Generationcited by 1
  13. 2026Eigenism: Ethics for a Human-AI Futurecited by 1
  14. 2025Splits! Flexible Sociocultural Linguistic Investigation at Scalecited by 0
  15. 2025From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Modelscited by 0
  16. 2025Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributionscited by 0
  17. 2025Hypernetworks for Perspectivist Adaptationcited by 0
  18. 2025Examining the Utility of Self-disclosure Types for Modeling Annotators of Social Normscited by 0
  19. 2026Learning Who Disagrees: Demographic Importance Weighting for Modeling Annotator Distributions with DiADEMcited by 0
  20. 2026IYKYK (But AI Doesn't): Automated Content Moderation Does Not Capture Communities' Heterogeneous Attitudes Towards Reclaimed Languagecited by 0
  21. 2026Who Decides What Is Harmful? Content Moderation Policy Through A Multi-Agent Personalised Inference Frameworkcited by 0
  22. 2026Temporal Simultaneity Predicts Annotation Quality in Sentiment Corporacited by 0
  23. 2026Evaluating Pluralism in LLMs through Latent Perspectivescited by 0
  24. 2026How Ethos and Pathos Appeals Resonate in Reader Interpretations of Social Media Messagescited by 0
  25. 2026LLMs as Oracles: Reliance on LLMs for Subjective Personal Questionscited by 0
  26. 2026Activating the Annotator: Operationalising Lived Experience in Data Annotationcited by 0