# Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers

arXiv: https://arxiv.org/abs/2309.12570 · HTML: https://mlsanity.com/paper/2309.12570/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 72 references that are in the map.
- First posted on arXiv 22 Sep 2023.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Enabling Large Language Models to Generate Text with Citations](https://mlsanity.com/paper/2305.14627/) (2023)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs](https://mlsanity.com/paper/2210.13312/) (2022)
- [Re3: Generating Longer Stories With Recursive Reprompting and Revision](https://mlsanity.com/paper/2210.06774/) (2022)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models](https://mlsanity.com/paper/2305.14763/) (2023)
- [Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker](https://mlsanity.com/paper/2306.00924/) (2023)
- [Evaluating Human-Language Model Interaction](https://mlsanity.com/paper/2212.09746/) (2022)
- [Sparks: Inspiration for Science Writing using Language Models](https://mlsanity.com/paper/2110.07640/) (2021)
- [Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers](https://mlsanity.com/paper/2211.05030/) (2022)
- [The Impact of Multiple Parallel Phrase Suggestions on Email Input and Composition Behaviour of Native and Non-Native English Writers](https://mlsanity.com/paper/2101.09157/) (2021)
- [Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works](https://mlsanity.com/paper/2210.08477/) (2022)
- [STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation](https://mlsanity.com/paper/2010.01717/) (2020)
- [Smart Reply: Automated Response Suggestion for Email](https://mlsanity.com/paper/1606.04870/) (2016)
- [Interacting with next-phrase suggestions: How suggestion systems aid and influence the cognitive processes of writing](https://mlsanity.com/paper/2208.00636/) (2022)
- [Citation: A Key to Building Responsible and Accountable Large Language Models](https://mlsanity.com/paper/2307.02185/) (2023)
- [CharacterChat: Supporting the Creation of Fictional Characters through Conversation and Progressive Manifestation with a Chatbot](https://mlsanity.com/paper/2106.12314/) (2021)
- [Mechanical Novel: Crowdsourcing Complex Work through Reflection and Revision](https://mlsanity.com/paper/1611.02682/) (2016)

## Built on by (most cited first, 30 of 32)

- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challenges](https://mlsanity.com/paper/2409.02387/) (2024)
- [AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://mlsanity.com/paper/2410.04265/) (2024)
- [Benchmarking Language Model Creativity: A Case Study on Code Generation](https://mlsanity.com/paper/2407.09007/) (2024)
- [MacGyver: Are Large Language Models Creative Problem Solvers?](https://mlsanity.com/paper/2311.09682/) (2023)
- [Divergent Creativity in Humans and Large Language Models](https://mlsanity.com/paper/2405.13012/) (2024)
- [Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications](https://mlsanity.com/paper/2409.16605/) (2024)
- [Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wild](https://mlsanity.com/paper/2505.16023/) (2025)
- [Creativity in LLM-based Multi-Agent Systems: A Survey](https://mlsanity.com/paper/2505.21116/) (2025)
- [Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers](https://mlsanity.com/paper/2403.01061/) (2024)
- [Guided Profile Generation Improves Personalization with LLMs](https://mlsanity.com/paper/2409.13093/) (2024)
- [User Modeling in the Era of Large Language Models: Current Research and Future Directions](https://mlsanity.com/paper/2312.11518/) (2023)
- [Large Language Model Benchmarks in Medical Tasks](https://mlsanity.com/paper/2410.21348/) (2024)
- [ProPerSim: Developing Proactive and Personalized AI Assistants through User-Assistant Simulation](https://mlsanity.com/paper/2509.21730/) (2025)
- [GPT for Games: A Scoping Review (2020-2023)](https://mlsanity.com/paper/2404.17794/) (2024)
- [The Impact and Opportunities of Generative AI in Fact-Checking](https://mlsanity.com/paper/2405.15985/) (2024)
- [GPT for Games: An Updated Scoping Review (2020-2024)](https://mlsanity.com/paper/2411.00308/) (2024)
- [Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models](https://mlsanity.com/paper/2505.16245/) (2025)
- [Creativity in AI: Progresses and Challenges](https://mlsanity.com/paper/2410.17218/) (2024)
- [Language Models as Critical Thinking Tools: A Case Study of Philosophers](https://mlsanity.com/paper/2404.04516/) (2024)
- [Investigating Wit, Creativity, and Detectability of Large Language Models in Domain-Specific Writing Style Adaptation of Reddit's Showerthoughts](https://mlsanity.com/paper/2405.01660/) (2024)
- [R2: A LLM Based Novel-to-Screenplay Generation Framework with Causal Plot Graphs](https://mlsanity.com/paper/2503.15655/) (2025)
- [Defining a New NLP Playground](https://mlsanity.com/paper/2310.20633/) (2023)
- [Randomness Is All You Need: Semantic Traversal of Problem-Solution Spaces with Large Language Models](https://mlsanity.com/paper/2402.06053/) (2024)
- [Mentigo: An Intelligent Agent for Mentoring Students in the Creative Problem Solving Process](https://mlsanity.com/paper/2409.14228/) (2024)
- [Societal Impacts Research Requires Benchmarks for Creative Composition Tasks](https://mlsanity.com/paper/2504.06549/) (2025)
- [Foundational Moral Values for AI Alignment](https://mlsanity.com/paper/2311.17017/) (2023)
- [Exploring AI Text Generation, Retrieval-Augmented Generation, and Detection Technologies: a Comprehensive Overview](https://mlsanity.com/paper/2412.03933/) (2024)
- [Initial Development and Evaluation of the Creative Artificial Intelligence through Recurring Developments and Determinations (CAIRDD) System](https://mlsanity.com/paper/2409.02291/) (2024)

## Abstract (arXiv)

The development of large language models (LLMs) capable of following instructions and engaging in conversational interactions sparked increased interest in their utilization across various support tools. We investigate the utility of modern LLMs in assisting professional writers via an empirical user study (n=30). The design of our collaborative writing interface is grounded in the cognitive process model of writing that views writing as a goal-oriented thinking process encompassing non-linear cognitive activities: planning, translating, and reviewing. Participants are asked to submit a post-completion survey to provide feedback on the potential and pitfalls of LLMs as writing collaborators. Upon analyzing the writer-LLM interactions, we find that while writers seek LLM's help across all three types of cognitive activities, they find LLMs more helpful in translation and reviewing. Our findings from analyzing both the interactions and the survey responses highlight future research directions in creative writing assistance using LLMs.