mlsanityData to 14 Oct 2026

CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation

arXiv 2206.08931 ↗First posted 9 Jun 2022cs.HC cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

Human annotated data plays a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into dataset annotation have not received nearly enough attention. In this paper, we survey an array of literature that provides insights into ethical considerations around crowdsourced dataset annotation. We synthesize these insights, and lay out the challenges in this space along two layers: (1) who the annotator is, and how the annotators' lived experiences can impact their annotations, and (2) the relationship between the annotators and the crowdsourcing platforms, and what that relationship affords them. Finally, we introduce a novel framework, CrowdWorkSheets, for dataset developers to facilitate transparent documentation of key decisions points at various stages of the data annotation pipeline: task formulation, selection of annotators, platform and infrastructure choices, dataset analysis and evaluation, and dataset release and maintenance.

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 17 of its 82 references that are in the map. First posted on arXiv 9 Jun 2022.

Builds on

17 of 82 references.

References · 17

  1. 2018Datasheets for Datasetscited by 1,386
  2. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 171
  3. 2020Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructurecited by 138
  4. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
  5. 2021Re-imagining Algorithmic Fairness in India and Beyondcited by 104
  6. 2018The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standardscited by 97
  7. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  8. 2019Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?cited by 64
  9. 2021Beyond Fair Pay: Ethical Implications of NLP Crowdsourcingcited by 53
  10. 2020Bringing the People Back In: Contesting Benchmark Machine Learning Datasetscited by 43
  11. 2017A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turkcited by 39
  12. 2021Detecting Cross-Geographic Biases in Toxicity Modeling on Social Mediacited by 31
5 more
  1. 2022The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligencecited by 26
  2. 2020Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Visioncited by 25
  3. 2021Quantifying the Invisible Labor in Crowd Workcited by 16
  4. 2018Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographicscited by 10
  5. 2021On the state of reporting in crowdsourcing experiments and a checklist to aid current practicescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 17; 2024: 20; 2025: 12; 2026: 5.

59 papers.

Sort:

By area:

  1. 2023Segment Anythingcited by 8,234
  2. 2023PaLM 2 Technical Reportcited by 1,162
  3. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 92
  4. 2023Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Promptingcited by 81
  5. 2023SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Modelscited by 66
  6. 2023FACET: Fairness in Computer Vision Evaluation Benchmarkcited by 58
  7. 2023AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asiacited by 54
  8. 2023"HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social mediacited by 45
  9. 2022Eliciting and Learning with Soft Labels from Every Annotatorcited by 43
  10. 2024If in a Crowdsourced Data Annotation Pipeline, a GPT-4cited by 38
  11. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  12. 2023Disentangling Perceptions of Offensiveness: Cultural and Moral Correlatescited by 26
38 more
  1. 2023Consensus and Subjectivity of Skin Tone Annotation for ML Fairnesscited by 22
  2. 2022Cultural Incongruencies in Artificial Intelligencecited by 21
  3. 2024Real Risks of Fake Data: Synthetic Data, Diversity-Washing and Consent Circumventioncited by 21
  4. 2025AI Alignment at Your Discretioncited by 20
  5. 2025The Case for "Thick Evaluations" of Cultural Representation in AIcited by 17
  6. 2024Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Wayscited by 16
  7. 2023Scaling Laws Do Not Scalecited by 13
  8. 2022A domain-specific language for describing machine learning datasetscited by 12
  9. 2023Going public: the role of public participation approaches in commercial AI labscited by 11
  10. 2024D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluationcited by 11
  11. 2024Blind Spots and Biases: Exploring the Role of Annotator Cognitive Biases in NLPcited by 11
  12. 2022Making Intelligence: Ethical Values in IQ and ML Benchmarkscited by 10
  13. 2024Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotationcited by 8
  14. 2023Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communitiescited by 6
  15. 2024Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Imagescited by 6
  16. 2024Copycats: the many lives of a publicly available medical imaging datasetcited by 6
  17. 2023Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collectioncited by 5
  18. 2023Is a Seat at the Table Enough? Engaging Teachers and Students in Dataset Specification for ML in Educationcited by 5
  19. 2024Racial/Ethnic Categories in AI and Algorithmic Fairness: Why They Matter and What They Representcited by 5
  20. 2025Decentralising LLM Alignment: A Case for Context, Pluralism, and Participationcited by 5
  21. 2024Surveys Considered Harmful? Reflecting on the Use of Surveys in AI Research, Development, and Governancecited by 4
  22. 2024What Makes An Expert? Reviewing How ML Researchers Define "Expert"cited by 4
  23. 2023The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practicescited by 3
  24. 2023SoUnD Framework: Analyzing (So)cial Representation in (Un)structured (D)atacited by 3
  25. 2024Promoting the Responsible Development of Speech Datasets for Mental Health and Neurological Disorders Researchcited by 3
  26. 2024Let Guidelines Guide You: A Prescriptive Guideline-Centered Data Annotation Methodologycited by 3
  27. 2025TechOps: Technical Documentation Templates for the AI Actcited by 3
  28. 2024Using Large Language Models to Enrich the Documentation of Datasets for Machine Learningcited by 2
  29. 2024A Taxonomy of Challenges to Curating Fair Datasetscited by 2
  30. 2025Talking About the Assumption in the Roomcited by 2
  31. 2026The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotationcited by 2
  32. 2022City-Wide Perceptions of Neighbourhood Quality using Street View Imagescited by 1
  33. 2024From Model Performance to Claim: How a Change of Focus in Machine Learning Replicability Can Help Bridge the Responsibility Gapcited by 1
  34. 2025An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footagecited by 1
  35. 2026From Reflection to Repair: A Scoping Review of Dataset Documentation Toolscited by 1
  36. 2024On the Readiness of Scientific Data for a Fair and Transparent Use in Machine Learningcited by 0
  37. 2024Documentation Practices of Artificial Intelligencecited by 0
  38. 2024Exploring the Capability of ChatGPT to Reproduce Human Labels for Social Computing Tasks (Extended Version)cited by 0

50 most cited of 59 All 59 (JSON)