# CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation

arXiv: https://arxiv.org/abs/2206.08931 · HTML: https://mlsanity.com/paper/2206.08931/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 82 references that are in the map.
- First posted on arXiv 9 Jun 2022.

## Builds on (most cited first, 17 of 17)

- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AI](https://mlsanity.com/paper/2204.01075/) (2022)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [On Releasing Annotator-Level Labels and Information in Datasets](https://mlsanity.com/paper/2110.05699/) (2021)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards](https://mlsanity.com/paper/1805.03677/) (2018)
- [Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development](https://mlsanity.com/paper/2108.04308/) (2021)
- [Garbage In, Garbage Out? Do Machine Learning Application Papers in Social Computing Report Where Human-Labeled Training Data Comes From?](https://mlsanity.com/paper/1912.08320/) (2019)
- [Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing](https://mlsanity.com/paper/2104.10097/) (2021)
- [Bringing the People Back In: Contesting Benchmark Machine Learning Datasets](https://mlsanity.com/paper/2007.07399/) (2020)
- [A Data-Driven Analysis of Workers' Earnings on Amazon Mechanical Turk](https://mlsanity.com/paper/1712.05796/) (2017)
- [Detecting Cross-Geographic Biases in Toxicity Modeling on Social Media](https://mlsanity.com/paper/2104.06999/) (2021)
- [The Dataset Nutrition Label (2nd Gen): Leveraging Context to Mitigate Harms in Artificial Intelligence](https://mlsanity.com/paper/2201.03954/) (2022)
- [Between Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision](https://mlsanity.com/paper/2007.14886/) (2020)
- [Quantifying the Invisible Labor in Crowd Work](https://mlsanity.com/paper/2110.00169/) (2021)
- [Characterizing the Global Crowd Workforce: A Cross-Country Comparison of Crowdworker Demographics](https://mlsanity.com/paper/1812.05948/) (2018)
- [On the state of reporting in crowdsourcing experiments and a checklist to aid current practices](https://mlsanity.com/paper/2107.13519/) (2021)

## Built on by (most cited first, 30 of 59)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023)
- [Sensitivity, Performance, Robustness: Deconstructing the Effect of Sociodemographic Prompting](https://mlsanity.com/paper/2309.07034/) (2023)
- [SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models](https://mlsanity.com/paper/2305.11840/) (2023)
- [FACET: Fairness in Computer Vision Evaluation Benchmark](https://mlsanity.com/paper/2309.00035/) (2023)
- [AI's Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia](https://mlsanity.com/paper/2305.11844/) (2023)
- ["HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media](https://mlsanity.com/paper/2304.10619/) (2023)
- [Eliciting and Learning with Soft Labels from Every Annotator](https://mlsanity.com/paper/2207.00810/) (2022)
- [If in a Crowdsourced Data Annotation Pipeline, a GPT-4](https://mlsanity.com/paper/2402.16795/) (2024)
- [Farsight: Fostering Responsible AI Awareness During AI Application Prototyping](https://mlsanity.com/paper/2402.15350/) (2024)
- [Disentangling Perceptions of Offensiveness: Cultural and Moral Correlates](https://mlsanity.com/paper/2312.06861/) (2023)
- [Consensus and Subjectivity of Skin Tone Annotation for ML Fairness](https://mlsanity.com/paper/2305.09073/) (2023)
- [Cultural Incongruencies in Artificial Intelligence](https://mlsanity.com/paper/2211.13069/) (2022)
- [Real Risks of Fake Data: Synthetic Data, Diversity-Washing and Consent Circumvention](https://mlsanity.com/paper/2405.01820/) (2024)
- [AI Alignment at Your Discretion](https://mlsanity.com/paper/2502.10441/) (2025)
- [The Case for "Thick Evaluations" of Cultural Representation in AI](https://mlsanity.com/paper/2503.19075/) (2025)
- [Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways](https://mlsanity.com/paper/2406.11980/) (2024)
- [Scaling Laws Do Not Scale](https://mlsanity.com/paper/2307.03201/) (2023)
- [A domain-specific language for describing machine learning datasets](https://mlsanity.com/paper/2207.02848/) (2022)
- [Going public: the role of public participation approaches in commercial AI labs](https://mlsanity.com/paper/2306.09871/) (2023)
- [D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation](https://mlsanity.com/paper/2404.10857/) (2024)
- [Blind Spots and Biases: Exploring the Role of Annotator Cognitive Biases in NLP](https://mlsanity.com/paper/2404.19071/) (2024)
- [Making Intelligence: Ethical Values in IQ and ML Benchmarks](https://mlsanity.com/paper/2209.00692/) (2022)
- [Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotation](https://mlsanity.com/paper/2402.06811/) (2024)
- [Modeling subjectivity (by Mimicking Annotator Annotation) in toxic comment identification across diverse communities](https://mlsanity.com/paper/2311.00203/) (2023)
- [Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Images](https://mlsanity.com/paper/2401.14322/) (2024)
- [Copycats: the many lives of a publicly available medical imaging dataset](https://mlsanity.com/paper/2402.06353/) (2024)
- [Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection](https://mlsanity.com/paper/2308.12885/) (2023)
- [Is a Seat at the Table Enough? Engaging Teachers and Students in Dataset Specification for ML in Education](https://mlsanity.com/paper/2311.05792/) (2023)

## Abstract (arXiv)

Human annotated data plays a crucial role in machine learning (ML) research and development. However, the ethical considerations around the processes and decisions that go into dataset annotation have not received nearly enough attention. In this paper, we survey an array of literature that provides insights into ethical considerations around crowdsourced dataset annotation. We synthesize these insights, and lay out the challenges in this space along two layers: (1) who the annotator is, and how the annotators' lived experiences can impact their annotations, and (2) the relationship between the annotators and the crowdsourcing platforms, and what that relationship affords them. Finally, we introduce a novel framework, CrowdWorkSheets, for dataset developers to facilitate transparent documentation of key decisions points at various stages of the data annotation pipeline: task formulation, selection of annotators, platform and infrastructure choices, dataset analysis and evaluation, and dataset release and maintenance.