# Quantifying the Invisible Labor in Crowd Work

arXiv: https://arxiv.org/abs/2110.00169 · HTML: https://mlsanity.com/paper/2110.00169/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Oct 2021.

## Built on by (most cited first, 16 of 16)

- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation](https://mlsanity.com/paper/2206.08931/) (2022)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://mlsanity.com/paper/2305.00955/) (2023)
- [Ethical Considerations for Responsible Data Curation](https://mlsanity.com/paper/2302.03629/) (2023)
- [Why Does ChatGPT "Delve" So Much? Exploring the Sources of Lexical Overrepresentation in Large Language Models](https://mlsanity.com/paper/2412.11385/) (2024)
- [Fairshare Data Pricing via Data Valuation for Large Language Models](https://mlsanity.com/paper/2502.00198/) (2025)
- [The ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices](https://mlsanity.com/paper/2306.01800/) (2023)
- [Making Data Work Count](https://mlsanity.com/paper/2311.18046/) (2023)
- [A Taxonomy of Challenges to Curating Fair Datasets](https://mlsanity.com/paper/2406.06407/) (2024)
- [Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback](https://mlsanity.com/paper/2508.01930/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Ethics Whitepaper: Whitepaper on Ethical Research into Large Language Models](https://mlsanity.com/paper/2410.19812/) (2024)
- [A Culturally-Aware Tool for Crowdworkers: Leveraging Chronemics to Support Diverse Work Styles](https://mlsanity.com/paper/2408.07838/) (2024)
- [Data Enrichment Work and AI Labor in Latin America and the Caribbean](https://mlsanity.com/paper/2501.06981/) (2025)
- [Upskilling with Generative AI: Practices and Challenges for Freelance Knowledge Workers](https://mlsanity.com/paper/2604.27231/) (2026)
- [How Formerly Incarcerated People Envision Technologies for Prison Parole](https://mlsanity.com/paper/2607.16513/) (2026)

## Abstract (arXiv)

Crowdsourcing markets provide workers with a centralized place to find paid work. What may not be obvious at first glance is that, in addition to the work they do for pay, crowd workers also have to shoulder a variety of unpaid invisible labor in these markets, which ultimately reduces workers' hourly wages. Invisible labor includes finding good tasks, messaging requesters, or managing payments. However, we currently know little about how much time crowd workers actually spend on invisible labor or how much it costs them economically. To ensure a fair and equitable future for crowd work, we need to be certain that workers are being paid fairly for all of the work they do. In this paper, we conduct a field study to quantify the invisible labor in crowd work. We build a plugin to record the amount of time that 100 workers on Amazon Mechanical Turk dedicate to invisible labor while completing 40,903 tasks. If we ignore the time workers spent on invisible labor, workers' median hourly wage was 3.76. But, we estimated that crowd workers in our study spent 33% of their time daily on invisible labor, dropping their median hourly wage to 2.83. We found that the invisible labor differentially impacts workers depending on their skill level and workers' demographics. The invisible labor category that took the most time and that was also the most common revolved around workers having to manage their payments. The second most time-consuming invisible labor category involved hyper-vigilance, where workers vigilantly watched over requesters' profiles for newly posted work or vigilantly searched for labor. We hope that through our paper, the invisible labor in crowdsourcing becomes more visible, and our results help to reveal the larger implications of the continuing invisibility of labor in crowdsourcing.