mlsanityData to 14 Oct 2026

When Do Annotator Demographics Matter? Measuring the Influence of Annotator Demographics with the POPQUORN Dataset

arXiv 2306.06826 ↗First posted 12 Jun 2023cs.CL cs.AI cs.CYcited by 52

Not on the map. Why?

Abstract (arXiv)

Annotators are not fungible. Their demographics, life experiences, and backgrounds all contribute to how they label data. However, NLP has only recently considered how annotator identity might influence their decisions. Here, we present POPQUORN (the POtato-Prolific dataset for QUestion-Answering, Offensiveness, text Rewriting, and politeness rating with demographic Nuance). POPQUORN contains 45,000 annotations from 1,484 annotators, drawn from a representative sample regarding sex, age, and race as the US population. Through a series of analyses, we show that annotators' background plays a significant role in their judgments. Further, our work shows that backgrounds not previously considered in NLP (e.g., education), are meaningful and should be considered. Our study suggests that understanding the background of annotators and collecting labels from a demographically balanced pool of crowd workers is important to reduce the bias of datasets. The dataset, annotator background, and annotation interface are available at https://github.com/Jiaxin-Pei/potato-prolific-dataset .

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 23 of its 55 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

23 of 55 references.

References · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  7. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  8. 2023Data-centric Artificial Intelligence: A Surveycited by 203
  9. 2020Directions in Abusive Language Training Data: Garbage In, Garbage Outcited by 140
  10. 2018A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspectivecited by 112
  11. 2013A Computational Approach to Politeness with Application to Social Factorscited by 106
  12. 2021QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehensioncited by 91
11 more
  1. 2020Politeness Transfer: A Tag and Generate Approachcited by 78
  2. 2020ConvoKit: A Toolkit for the Analysis of Conversationscited by 72
  3. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 72
  4. 2021Towards generalisable hate speech detection: a review on obstacles and solutionscited by 66
  5. 2022POTATO: The Portable Text Annotation Toolcited by 64
  6. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  7. 2017Detecting Hate Speech in Social Mediacited by 42
  8. 2021Ruddit: Norms of Offensiveness for English Reddit Commentscited by 27
  9. 2021Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communicationscited by 27
  10. 2022TyDiP: A Dataset for Politeness Classification in Nine Typologically Diverse Languagescited by 21
  11. 2023On the rise of fear speech in online social mediacited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 12; 2025: 22; 2026: 13.

52 papers.

Sort:

By area:

  1. 2024Quantifying the Persona Effect in LLM Simulationscited by 169
  2. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  3. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  4. 2023Exploring Cross-Cultural Differences in English Hate Speech Annotations: From Dataset Construction to Analysiscited by 45
  5. 2023Sociodemographic Prompting is Not Yet an Effective Approach for Simulating Subjective Judgments with LLMscited by 36
  6. 2025Beyond Demographics: Fine-tuning Large Language Models to Predict Individuals' Subjective Text Perceptionscited by 33
  7. 2024Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Artcited by 26
  8. 2024Personas with Attitudes: Controlling LLMs for Diverse Data Annotationcited by 21
  9. 2024Prompts Are Programs Too! Understanding How Developers Build Software Containing Promptscited by 19
  10. 2023GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectivescited by 16
  11. 2024D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluationcited by 11
  12. 2025Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Modelscited by 11
38 more
  1. 2024Which Demographics do LLMs Default to During Annotation?cited by 10
  2. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
  3. 2023Gen-Z: Generative Zero-Shot Text Classification with Contextualized Label Descriptionscited by 9
  4. 2025No for Some, Yes for Others: Persona Prompts and Other Sources of False Refusal in Language Modelscited by 9
  5. 2024Rater Cohesion and Quality from a Vicarious Perspectivecited by 8
  6. 2024Re-examining Sexism and Misogyny Classification with Annotator Attitudescited by 8
  7. 2024Robustness and Confounders in the Demographic Alignment of LLMs with Human Perceptions of Offensivenesscited by 7
  8. 2025NUTMEG: Separating Signal From Noise in Annotator Disagreementcited by 7
  9. 2024The Unseen Targets of Hate -- A Systematic Review of Hateful Communication Datasetscited by 6
  10. 2025Evaluating how LLM annotations represent diverse views on contentious topicscited by 6
  11. 2024Beyond Accuracy: Investigating Error Types in GPT-4 Responses to USMLE Questionscited by 5
  12. 2025Social Bias in Popular Question-Answering Benchmarkscited by 5
  13. 2025CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Modelscited by 5
  14. 2025Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanationcited by 4
  15. 2025Isolating Culture Neurons in Multilingual Large Language Modelscited by 4
  16. 2025Attention to Non-Adopterscited by 4
  17. 2025Ontologies in Design: How Imagining a Tree Reveals Possibilites and Assumptions in Large Language Modelscited by 3
  18. 2025Hope vs. Hate: Understanding User Interactions with LGBTQ+ News Content in Mainstream US News Media through the Lens of Hope Speechcited by 2
  19. 2025Conflicts in Texts: Data, Implications and Challengescited by 2
  20. 2025Improving the Distributional Alignment of LLMs using Supervisioncited by 2
  21. 2025Distributions In, Distributions Out: The Case for Soft-Label Trainingcited by 2
  22. 2026From Ground Truth to Measurement: A Statistical Framework for Human Labelingcited by 2
  23. 2025Aligning NLP Models with Target Population Perspectives using PAIR: Population-Aligned Instance Replicationcited by 1
  24. 2025Let Them Down Easy! Contextual Effects of LLM Guardrails on User Perceptions and Preferencescited by 1
  25. 2025ModelCitizens: Representing Community Voices in Online Safetycited by 1
  26. 2026Multi-Perspective LLM Annotations for Valid Analyses in Subjective Taskscited by 1
  27. 2026Who and What? Using Linguistic Features and Annotator Characteristics to Analyze Annotation Variationcited by 1
  28. 2026P1SCO: Social Dimensions from a Perspectivist Lenscited by 1
  29. 2026Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025cited by 1
  30. 2025Are Humans as Brittle as Large Language Models?cited by 0
  31. 2025Balancing Quality and Variation: Spam Filtering Distorts Data Label Distributionscited by 0
  32. 2025Examining the Utility of Self-disclosure Types for Modeling Annotators of Social Normscited by 0
  33. 2026Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLPcited by 0
  34. 2026Learning Who Disagrees: Demographic Importance Weighting for Modeling Annotator Distributions with DiADEMcited by 0
  35. 2026IYKYK (But AI Doesn't): Automated Content Moderation Does Not Capture Communities' Heterogeneous Attitudes Towards Reclaimed Languagecited by 0
  36. 2026When Does Demographic Information Help? Data and Modeling Regimes for Perspective-Aware Hate Speech Detectioncited by 0
  37. 2026Learning Perspectivist Social Meaning via Demographic-Conditioned Fusion Embeddingscited by 0
  38. 2026Thesis Proposal: Toward a Human-Centered and Perspective-Aware Framework for Reproducible ML Evaluation and AI Alignmentcited by 0

50 most cited of 52 All 52 (JSON)