mlsanityData to 7 Oct 2026

Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks

arXiv 2306.07899 ↗First posted 13 Jun 2023cs.CL cs.CYcited by 102

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are remarkable data annotators. They can be used to generate high-fidelity supervised training data, as well as survey and experimental data. With the widespread adoption of LLMs, human gold--standard annotations are key to understanding the capabilities of LLMs and the validity of their results. However, crowdsourcing, an important, inexpensive way to obtain human annotations, may itself be impacted by LLMs, as crowd workers have financial incentives to use LLMs to increase their productivity and income. To investigate this concern, we conducted a case study on the prevalence of LLM usage by crowd workers. We reran an abstract summarization task from the literature on Amazon Mechanical Turk and, through a combination of keystroke detection and synthetic text classification, estimate that 33-46% of crowd workers used LLMs when completing the task. Although generalization to other, less LLM-friendly tasks is unclear, our results call for platforms, researchers, and crowd workers to find new ways to ensure that human data remain human, perhaps using the methodology proposed here as a stepping stone. Code/data: https://github.com/epfl-dlab/GPTurk

Key facts. Cited by 102 papers in the map (data through 7 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 13 Jun 2023.

Builds on

23 of 46 references.

References · 23

  1. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  2. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  3. 2023Whose Opinions Do Language Models Reflect?cited by 759
  4. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 737
  5. 2023A Watermark for Large Language Modelscited by 667
  6. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  7. 2023Can AI-Generated Text be Reliably Detected?cited by 372
  8. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  9. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 259
  10. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  11. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  12. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
11 more
  1. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  2. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  3. 2022Human heuristics for AI-generated language are flawedcited by 104
  4. 2023ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learningcited by 97
  5. 2023Ghostbuster: Detecting Text Ghostwritten by Large Language Modelscited by 90
  6. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  7. 2023CHEAT: A Large-scale Dataset for Detecting ChatGPT-writtEn AbsTractscited by 39
  8. 2023Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Sciencecited by 37
  9. 2021Fingerprinting Fine-tuned Language Models in the Wildcited by 9
  10. 2023string2string: A Modern Python Library for String-to-String Algorithmscited by 9
  11. 2019Message Distortion in Information Cascadescited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 32; 2024: 42; 2025: 21; 2026: 7.

102 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  3. 2023Self-Consuming Generative Models Go MADcited by 239
  4. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  5. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 154
  6. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 127
  7. 2024Mapping the Increasing Use of LLMs in Scientific Paperscited by 112
  8. 2023A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directionscited by 101
  9. 2025LLM Social Simulations Are a Promising Research Methodcited by 97
  10. 2023CoMPosT: Characterizing and Evaluating Caricature in LLM Simulationscited by 94
  11. 2023Large Language Models Suffer From Their Own Output: An Analysis of the Self-Consuming Training Loopcited by 75
  12. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
38 more
  1. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  2. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
  3. 2024Watermarking Makes Language Models Radioactivecited by 52
  4. 2024Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Modelscited by 51
  5. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  6. 2024Evaluating Language Model Agency through Negotiationscited by 43
  7. 2025Infrastructure for AI Agentscited by 40
  8. 2024Network Formation and Dynamics Among Multi-LLMscited by 28
  9. 2024AFaCTA: Assisting the Annotation of Factual Claim Detection with Reliable LLM Annotatorscited by 28
  10. 2023LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMscited by 27
  11. 2023Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytellingcited by 26
  12. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  13. 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
  14. 2024"They are uncultured": Unveiling Covert Harms and Social Threats in LLM Generated Conversationscited by 24
  15. 2024A Comparative Study on Annotation Quality of Crowdsourcing and LLM via Label Aggregationcited by 23
  16. 2023Are Large Language Models a Threat to Digital Public Goods? Evidence from Activity on Stack Overflowcited by 22
  17. 2023Using Large Language Models for Qualitative Analysis can Introduce Serious Biascited by 22
  18. 2024Adaptive Ensembles of Fine-Tuned Transformers for LLM-Generated Text Detectioncited by 22
  19. 2024MemeGuard: An LLM and VLM-based Framework for Advancing Content Moderation via Meme Interventioncited by 20
  20. 2024AI-Assisted Human Evaluation of Machine Translationcited by 20
  21. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  22. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  23. 2024Understanding Figurative Meaning through Explainable Visual Entailmentcited by 19
  24. 2023ParaGuide: Guided Diffusion Paraphrasers for Plug-and-Play Textual Style Transfercited by 18
  25. 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 17
  26. 2024ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?cited by 17
  27. 2024Data Authenticity, Consent, & Provenance for AI are all broken: what will it take to fix them?cited by 15
  28. 2024Regurgitative Training: The Value of Real Data in Training Large Language Modelscited by 15
  29. 2023Leveraging Implicit Feedback from Deployment Data in Dialoguecited by 14
  30. 2023A Large Language Model Approach to Educational Survey Feedback Analysiscited by 14
  31. 2023Next Steps for Human-Centered Generative AI: A Technical Perspectivecited by 13
  32. 2024Faithful Temporal Question Answering over Heterogeneous Sourcescited by 13
  33. 2023Revisiting Prompt Engineering via Declarative Crowdsourcingcited by 12
  34. 2023ALMANACS: A Simulatability Benchmark for Language Model Explainabilitycited by 11
  35. 2024AmbigNLG: Addressing Task Ambiguity in Instruction for NLGcited by 10
  36. 2024MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasetscited by 10
  37. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  38. 2025textttDroid: A Resource Suite for AI-Generated Code Detectioncited by 9

50 most cited of 102 All 102 (JSON)