# Prevalence and prevention of large language model use in crowd work

arXiv: https://arxiv.org/abs/2310.15683 · HTML: https://mlsanity.com/paper/2310.15683/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 9 of 9)

- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [Reward Collapse in Aligning Large Language Models](https://mlsanity.com/paper/2305.17608/) (2023)
- [Message Distortion in Information Cascades](https://mlsanity.com/paper/1902.09197/) (2019)

## Built on by (most cited first, 1 of 1)

- [LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arena](https://mlsanity.com/paper/2501.03266/) (2025)

## Abstract (arXiv)

We show that the use of large language models (LLMs) is prevalent among crowd workers, and that targeted mitigation strategies can significantly reduce, but not eliminate, LLM use. On a text summarization task where workers were not directed in any way regarding their LLM use, the estimated prevalence of LLM use was around 30%, but was reduced by about half by asking workers to not use LLMs and by raising the cost of using them, e.g., by disabling copy-pasting. Secondary analyses give further insight into LLM use and its prevention: LLM use yields high-quality but homogeneous responses, which may harm research concerned with human (rather than model) behavior and degrade future models trained with crowdsourced data. At the same time, preventing LLM use may be at odds with obtaining high-quality responses; e.g., when requesting workers not to use LLMs, summaries contained fewer keywords carrying essential information. Our estimates will likely change as LLMs increase in popularity or capabilities, and as norms around their usage change. Yet, understanding the co-evolution of LLM-based tools and users is key to maintaining the validity of research done using crowdsourcing, and we provide a critical baseline before widespread adoption ensues.