mlsanityData to 7 Oct 2026

Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing

arXiv 2104.10097 ↗First posted 20 Apr 2021cs.CL cs.CYcited by 52

Not on the map. Why?

Abstract (arXiv)

The use of crowdworkers in NLP research is growing rapidly, in tandem with the exponential increase in research production in machine learning and AI. Ethical discussion regarding the use of crowdworkers within the NLP research community is typically confined in scope to issues related to labor conditions such as fair pay. We draw attention to the lack of ethical considerations related to the various tasks performed by workers, including labeling, evaluation, and production. We find that the Final Rule, the common ethical framework used by researchers, did not anticipate the use of online crowdsourcing platforms for data collection, resulting in gaps between the spirit and practice of human-subjects ethics in NLP research. We enumerate common scenarios where crowdworkers performing NLP tasks are at risk of harm. We thus recommend that researchers evaluate these risks by considering the three ethical principles set up by the Belmont Report. We also clarify some common misconceptions regarding the Institutional Review Board (IRB) application. We hope this paper will serve to reopen the discussion within our community regarding the ethical use of crowdworkers.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 6 of its 65 references that are in the map. First posted on arXiv 20 Apr 2021.

Builds on

6 of 65 references.

References · 6

  1. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 703
  2. 2019Social Bias Frames: Reasoning about Social and Power Implications of Languagecited by 377
  3. 2020A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Supportcited by 182
  4. 2019Learning to Speak and Act in a Fantasy Text Adventure Gamecited by 122
  5. 2016Conversational Markers of Constructive Discussionscited by 17
  6. 2016Personalized Emphasis Framing for Persuasive Message Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 12; 2023: 16; 2024: 10; 2025: 3; 2026: 2.

52 papers.

Sort:

By area:

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  3. 2022POTATO: The Portable Text Annotation Toolcited by 64
  4. 2022CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotationcited by 59
  5. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  6. 2021Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotationcited by 51
  7. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  8. 2021ConvAbuse: Data, Analysis, and Benchmarks for Nuanced Abuse Detection in Conversational AIcited by 46
  9. 2023Analyzing Dataset Annotation Quality Management in the Wildcited by 39
  10. 2023Quantifying the Dialect Gap and its Correlates Across Languagescited by 33
  11. 2022Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamicscited by 26
  12. 2024Position: Measure Dataset Diversity, Don't Just Claim Itcited by 26
38 more
  1. 2022Bugs in the Data: How ImageNet Misrepresents Biodiversitycited by 24
  2. 2021Ethics Sheet for Automatic Emotion Recognition and Sentiment Analysiscited by 22
  3. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 22
  4. 2022Listening to Affected Communities to Define Extreme Speech: Dataset and Experimentscited by 21
  5. 2021A Review of Human Evaluation for Style Transfercited by 19
  6. 2023A Material Lens on Coloniality in NLPcited by 19
  7. 2023Ethical Considerations for Responsible Data Curationcited by 17
  8. 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 17
  9. 2023Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSAcited by 15
  10. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  11. 2023Are Language Models Worse than Humans at Following Prompts? It's Complicatedcited by 14
  12. 2024Exploring Musical Roots: Applying Audio Embeddings to Empower Influence Attribution for a Generative Music Modelcited by 13
  13. 2021Quantifying and Avoiding Unfair Qualification Labour in Crowdsourcingcited by 11
  14. 2022Risk-graded Safety for Handling Medical Queries in Conversational AIcited by 11
  15. 2023The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generationcited by 11
  16. 2026Counting on Consensus: Selecting the Right Inter-annotator Agreement Metric for NLP Annotation and Evaluationcited by 11
  17. 2024FAIR Enough: How Can We Develop and Assess a FAIR-Compliant Dataset for Large Language Models' Training?cited by 9
  18. 2024Voices in a Crowd: Searching for Clusters of Unique Perspectivescited by 9
  19. 2024Re-examining Sexism and Misogyny Classification with Annotator Attitudescited by 7
  20. 2022ParaCotta: Synthetic Multilingual Paraphrase Corpora from the Most Diverse Translation Sample Paircited by 6
  21. 2023Thresh: A Unified, Customizable and Deployable Platform for Fine-Grained Text Evaluationcited by 6
  22. 2023Fairness and Bias in Truth Discovery Algorithms: An Experimental Analysiscited by 4
  23. 2023Lessons Learned from a Citizen Science Project for Natural Language Processingcited by 4
  24. 2021Use of Formal Ethical Reviews in NLP Literature: Historical Trends and Current Practicescited by 3
  25. 2023The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practicescited by 3
  26. 2024Promoting Constructive Deliberation: Reframing for Receptivenesscited by 3
  27. 2025Envisioning Stakeholder-Action Pairs to Mitigate Negative Impacts of AI: A Participatory Approach to Inform Policy Makingcited by 2
  28. 2025NLP Security and Ethics, in the Wildcited by 2
  29. 2021Measuring Ethics in AI with AI: A Methodology and Dataset Constructioncited by 1
  30. 2021Augmenting semantic lexicons using word embeddings and transfer learningcited by 1
  31. 2022Resolving the Human Subjects Status of Machine Learning's Crowdworkerscited by 1
  32. 2022Atomist or Holist? A Diagnosis and Vision for More Productive Interdisciplinary AI Ethics Dialoguecited by 1
  33. 2023Sociocultural knowledge is needed for selection of shots in hate speech detection taskscited by 1
  34. 2023On Degrees of Freedom in Defining and Testing Natural Language Understandingcited by 1
  35. 2024HarmPot: An Annotation Framework for Evaluating Offline Harm Potential of Social Media Textcited by 1
  36. 2024Beyond Generative Artificial Intelligence: Roadmap for Natural Language Generationcited by 1
  37. 2024Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Modelscited by 1
  38. 2022Contextualizing Artificially Intelligent Morality: A Meta-Ethnography of Top-Down, Bottom-Up, and Hybrid Models for Theoretical and Applied Ethics in Artificial Intelligencecited by 0

50 most cited of 52 All 52 (JSON)