Power-up! What Can Generative Models Do for Human Computation Workflows?
Not on the map. Why?
Abstract (arXiv)
We are amidst an explosion of artificial intelligence research, particularly around large language models (LLMs). These models have a range of applications across domains like medicine, finance, commonsense knowledge graphs, and crowdsourcing. Investigation into LLMs as part of crowdsourcing workflows remains an under-explored space. The crowdsourcing research community has produced a body of work investigating workflows and methods for managing complex tasks using hybrid human-AI methods. Within crowdsourcing, the role of LLMs can be envisioned as akin to a cog in a larger wheel of workflows. From an empirical standpoint, little is currently understood about how LLMs can improve the effectiveness of crowdsourcing workflows and how such workflows can be evaluated. In this work, we present a vision for exploring this gap from the perspectives of various stakeholders involved in the crowdsourcing paradigm -- the task requesters, crowd workers, platforms, and end-users. We identify junctures in typical crowdsourcing workflows at which the introduction of LLMs can play a beneficial role and propose means to augment existing design patterns for crowd work.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 49 references that are in the map. First posted on arXiv 5 Jul 2023.
Builds on
23 of 49 references.
References · 23
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2019Publicly Available Clinical BERT Embeddingscited by 995
- 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
- 2019Passage Re-ranking with BERTcited by 629
- 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
- 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
- 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
- 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 219
11 more
- 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
- 2020FinBERT: A Pretrained Language Model for Financial Communicationscited by 161
- 2022Calibrating Factual Knowledge in Pretrained Language Modelscited by 92
- 2021Sparks: Inspiration for Science Writing using Language Modelscited by 80
- 2019DocBERT: BERT for Document Classificationcited by 68
- 2019Patent Claim Generation by Fine-Tuning OpenAI GPT-2cited by 49
- 2022On the Explainability of Natural Language Processing Deep Modelscited by 37
- 2020Learning-to-Rank with BERT in TF-Rankingcited by 36
- 2021Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistantscited by 31
- 2022On the Compositional Generalization Gap of In-Context Learningcited by 24
- 2022Towards Process-Oriented, Modular, and Versatile Question Generation that Meets Educational Needscited by 19
Built on by
Citing papers per year: 2023: 1.
1 paper.