mlsanityData to 7 Oct 2026

CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities

arXiv 2201.06796 ↗First posted 18 Jan 2022cs.HC cs.CLcited by 245

Not on the map. Why?

Abstract (arXiv)

Large language models (LMs) offer unprecedented language generation capabilities and exciting opportunities for interaction design. However, their highly context-dependent capabilities are difficult to grasp and are often subjectively interpreted. In this paper, we argue that by curating and analyzing large interaction datasets, the HCI community can foster more incisive examinations of LMs' generative capabilities. Exemplifying this approach, we present CoAuthor, a dataset designed for revealing GPT-3's capabilities in assisting creative and argumentative writing. CoAuthor captures rich interactions between 63 writers and four instances of GPT-3 across 1445 writing sessions. We demonstrate that CoAuthor can address questions about GPT-3's language, ideation, and collaboration capabilities, and reveal its contribution as a writing "collaborator" under various definitions of good collaboration. Finally, we discuss how this work may facilitate a more principled discussion around LMs' promises and pitfalls in relation to interaction design. The dataset and an interface for replaying the writing sessions are publicly available at https://coauthor.stanford.edu.

Key facts. Cited by 245 papers in the map (data through 7 Oct 2026). Builds on 29 of its 87 references that are in the map. First posted on arXiv 18 Jan 2022.

Builds on

29 of 87 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  7. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  11. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  12. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
17 more
  1. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  2. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  3. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  4. 2020A Simple Language Model for Task-Oriented Dialoguecited by 331
  5. 2020Evaluation of Text Generation: A Surveycited by 275
  6. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  7. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  8. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  9. 2021Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processingcited by 126
  10. 2021The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writerscited by 61
  11. 2016Learning Language Games through Interactioncited by 43
  12. 2021Wordcraft: a Human-AI Collaborative Editor for Story Writingcited by 41
  13. 2017Naturalizing a Programming Language via Interactive Learningcited by 20
  14. 2021An Evaluation of Generative Pre-Training Model-based Therapy Chatbot for Caregiverscited by 15
  15. 2020On the Generation of Medical Dialogues for COVID-19cited by 12
  16. 2019Importance-Aware Learning for Neural Headline Editingcited by 3
  17. 2021Look at Me When I Talk to You: A Video Dataset to Enable Voice Assistants to Recognize Errorscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 61; 2024: 60; 2025: 51; 2026: 53.

245 papers.

Sort:

By area:

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  2. 2022Emergent Abilities of Large Language Modelscited by 2,483
  3. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  4. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  5. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  6. 2024LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contextscited by 240
  7. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  8. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  9. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  10. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  11. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  12. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
38 more
  1. 2023FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learningcited by 146
  2. 2023Co-Writing with Opinionated Language Models Affects Users' Viewscited by 145
  3. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  4. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
  5. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 99
  6. 2022Evaluating Human-Language Model Interactioncited by 90
  7. 2022PEER: A Collaborative Language Modelcited by 86
  8. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  9. 2023RecurrentGPT: Interactive Generation of (Arbitrarily) Long Textcited by 84
  10. 2024A Design Space for Intelligent and Interactive Writing Assistantscited by 76
  11. 2024Building Machines that Learn and Think with Peoplecited by 73
  12. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  13. 2023Evaluating Language Models for Mathematics through Interactionscited by 69
  14. 2022Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Supportcited by 68
  15. 2022Enabling Conversational Interaction with Mobile UI using Large Language Modelscited by 66
  16. 2022"Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interactioncited by 63
  17. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  18. 2024AI Suggestions Homogenize Writing Toward Western Styles and Diminish Cultural Nuancescited by 59
  19. 2023On the Creativity of Large Language Modelscited by 57
  20. 2024Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Modelscited by 51
  21. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  22. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 50
  23. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  24. 2023Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Datacited by 48
  25. 2022Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summariescited by 47
  26. 2022Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revisioncited by 46
  27. 2023VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototypingcited by 46
  28. 2024Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertisecited by 46
  29. 20223DALL-E: Integrating Text-to-Image AI in 3D Design Workflowscited by 41
  30. 2023Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Promptingcited by 36
  31. 2023Writer-Defined AI Personas for On-Demand Feedback Generationcited by 36
  32. 2023Anticipatory Music Transformercited by 35
  33. 2023Beyond the Chat: Executable and Verifiable Text-Editing with LLMscited by 35
  34. 2023ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviewscited by 34
  35. 2023CoQuest: Exploring Research Question Co-Creation with an LLM-based Agentcited by 34
  36. 2023Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writerscited by 32
  37. 2023Ecosystem Graphs: The Social Footprint of Foundation Modelscited by 31
  38. 2023ConvXAI: Delivering Heterogeneous AI Explanations via Conversations to Support Human-AI Scientific Writingcited by 31

50 most cited of 245 All 245 (JSON)