mlsanityData to 7 Oct 2026

On Releasing Annotator-Level Labels and Information in Datasets

arXiv 2110.05699 ↗First posted 12 Oct 2021cs.CL cs.CYcited by 123

Not on the map. Why?

Abstract (arXiv)

A common practice in building NLP datasets, especially using crowd-sourced annotations, involves obtaining multiple annotator judgements on the same data instances, which are then flattened to produce a single "ground truth" label or score, through majority voting, averaging, or adjudication. While these approaches may be appropriate in certain annotation tasks, such aggregations overlook the socially constructed nature of human perceptions that annotations for relatively more subjective tasks are meant to capture. In particular, systematic disagreements between annotators owing to their socio-cultural backgrounds and/or lived experiences are often obfuscated through such aggregations. In this paper, we empirically demonstrate that label aggregation may introduce representational biases of individual and group perspectives. Based on this finding, we propose a set of recommendations for increased utility and transparency of datasets for downstream use cases.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Builds on 7 of its 32 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

7 of 32 references.

References · 7

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 724
  3. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
  4. 2018Large Scale Crowdsourcing and Characterization of Twitter Abusive Behaviorcited by 265
  5. 2017Who Said What: Modeling Individual Labelers Improves Classificationcited by 77
  6. 2020Detecting Stance in Media on Global Warmingcited by 48
  7. 2015Statistical modality tagging from rule-based annotations and crowdsourcingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 21; 2023: 29; 2024: 33; 2025: 20; 2026: 15.

123 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  3. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  4. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  5. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  6. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  7. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  8. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
  9. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
  10. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 115
  11. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
  12. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
38 more
  1. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  2. 2023When the Majority is Wrong: Modeling Annotator Disagreement for Subjective Taskscited by 89
  3. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 80
  4. 2022Stop Measuring Calibration When Humans Disagreecited by 70
  5. 2023AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languagescited by 68
  6. 2023SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Modelscited by 66
  7. 2022CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotationcited by 59
  8. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  9. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  10. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  11. 2021Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotationcited by 51
  12. 2022Evaluation Gaps in Machine Learning Practicecited by 48
  13. 2023SemEval-2023 Task 10: Explainable Detection of Online Sexismcited by 48
  14. 2023The Ecological Fallacy in Annotation: Modelling Human Label Variation goes beyond Sociodemographicscited by 45
  15. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  16. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  17. 2024The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labelscited by 37
  18. 2024Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targetscited by 31
  19. 2024The Effectiveness of LLMs as Annotators: A Comparative Overview and Empirical Analysis of Direct Representationcited by 29
  20. 2023Capturing Perspectives of Crowdsourced Annotators in Subjective Learning Taskscited by 28
  21. 2024ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generationcited by 27
  22. 2023Disentangling Perceptions of Offensiveness: Cultural and Moral Correlatescited by 26
  23. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Datacited by 23
  24. 2022Unifying Data Perspectivism and Personalization: An Application to Social Normscited by 22
  25. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22
  26. 2022The Moral Foundations Reddit Corpuscited by 20
  27. 2023Automated Ableism: An Exploration of Explicit Disability Biases in Sentiment and Toxicity Analysis Modelscited by 20
  28. 2025AI Alignment at Your Discretioncited by 20
  29. 2023Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensivecited by 16
  30. 2023Representativeness as a Forgotten Lesson for Multilingual and Code-switched Data Collection and Preparationcited by 16
  31. 2023GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectivescited by 16
  32. 2022A Theory of Dynamic Benchmarkscited by 15
  33. 2022Toward More Meaningful Resources for Lower-resourced Languagescited by 14
  34. 2023Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Designcited by 14
  35. 2024Crowd-Calibrator: Can Annotator Disagreement Inform Calibration in Subjective Tasks?cited by 13
  36. 2021Hate Speech Classifiers Learn Human-Like Social Stereotypescited by 12
  37. 2021Two Contrasting Data Annotation Paradigms for Subjective NLP Taskscited by 12
  38. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12

50 most cited of 123 All 123 (JSON)