# Art or Artifice? Large Language Models and the False Promise of Creativity

arXiv: https://arxiv.org/abs/2309.14556 · HTML: https://mlsanity.com/paper/2309.14556/

## Key facts

- Cited by 172 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 83 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 23 of 23)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities](https://mlsanity.com/paper/2201.06796/) (2022)
- [Plan-And-Write: Towards Better Automatic Storytelling](https://mlsanity.com/paper/1811.05701/) (2018)
- [Re3: Generating Longer Stories With Recursive Reprompting and Revision](https://mlsanity.com/paper/2210.06774/) (2022)
- [Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionals](https://mlsanity.com/paper/2209.14958/) (2022)
- [Human-like Summarization Evaluation with ChatGPT](https://mlsanity.com/paper/2304.02554/) (2023)
- [PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Tracking](https://mlsanity.com/paper/2004.14967/) (2020)
- [Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Tasks](https://mlsanity.com/paper/2306.07899/) (2023)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020)
- [Large Language Models Enable Few-Shot Clustering](https://mlsanity.com/paper/2307.00524/) (2023)
- [Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writers](https://mlsanity.com/paper/2211.05030/) (2022)
- [LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond](https://mlsanity.com/paper/2305.14540/) (2023)
- [Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writers](https://mlsanity.com/paper/2309.12570/) (2023)
- [Collaborative Storytelling with Large-scale Neural Language Models](https://mlsanity.com/paper/2011.10208/) (2020)
- [Next Steps for Human-Centered Generative AI: A Technical Perspective](https://mlsanity.com/paper/2306.15774/) (2023)
- [Ambient Adventures: Teaching ChatGPT on Developing Complex Stories](https://mlsanity.com/paper/2308.01734/) (2023)

## Built on by (most cited first, 30 of 172)

- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [Homogenization Effects of Large Language Models on Human Creative Ideation](https://mlsanity.com/paper/2402.01536/) (2024)
- [The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text](https://mlsanity.com/paper/2311.09807/) (2023)
- [A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing](https://mlsanity.com/paper/2310.08433/) (2023)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [Safeguarding Large Language Models: A Survey](https://mlsanity.com/paper/2406.02622/) (2024)
- [LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Play](https://mlsanity.com/paper/2405.06373/) (2024)
- [Building Guardrails for Large Language Models](https://mlsanity.com/paper/2402.01822/) (2024)
- [Agents' Room: Narrative Generation through Multi-step Collaboration](https://mlsanity.com/paper/2410.02603/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://mlsanity.com/paper/2501.00273/) (2024)
- [AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text](https://mlsanity.com/paper/2410.04265/) (2024)
- [LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing](https://mlsanity.com/paper/2507.00769/) (2025)
- [Benchmarking Language Model Creativity: A Case Study on Code Generation](https://mlsanity.com/paper/2407.09007/) (2024)
- [People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text](https://mlsanity.com/paper/2501.15654/) (2025)
- [Exploring the Impact of Temperature on Large Language Models:Hot or Cold?](https://mlsanity.com/paper/2506.07295/) (2025)
- [Assessing and Understanding Creativity in Large Language Models](https://mlsanity.com/paper/2401.12491/) (2024)
- [Aligning Language Models with Demonstrated Feedback](https://mlsanity.com/paper/2406.00888/) (2024)
- [DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reports](https://mlsanity.com/paper/2601.08536/) (2026)
- [Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluation](https://mlsanity.com/paper/2405.13769/) (2024)
- [Divergent Creativity in Humans and Large Language Models](https://mlsanity.com/paper/2405.13012/) (2024)
- [A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedians](https://mlsanity.com/paper/2405.20956/) (2024)
- [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025)
- [Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?](https://mlsanity.com/paper/2407.01119/) (2024)
- [Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publications](https://mlsanity.com/paper/2409.16605/) (2024)
- [LLM Probability Concentration: How Alignment Shrinks the Generative Horizon](https://mlsanity.com/paper/2506.17871/) (2025)
- [AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation](https://mlsanity.com/paper/2504.07532/) (2025)
- [Creative Preference Optimization](https://mlsanity.com/paper/2505.14442/) (2025)
- [CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraints](https://mlsanity.com/paper/2410.04197/) (2024)

## Abstract (arXiv)

Researchers have argued that large language models (LLMs) exhibit high-quality writing capabilities from blogs to stories. However, evaluating objectively the creativity of a piece of writing is challenging. Inspired by the Torrance Test of Creative Thinking (TTCT), which measures creativity as a process, we use the Consensual Assessment Technique [3] and propose the Torrance Test of Creative Writing (TTCW) to evaluate creativity as a product. TTCW consists of 14 binary tests organized into the original dimensions of Fluency, Flexibility, Originality, and Elaboration. We recruit 10 creative writers and implement a human assessment of 48 stories written either by professional authors or LLMs using TTCW. Our analysis shows that LLM-generated stories pass 3-10X less TTCW tests than stories written by professionals. In addition, we explore the use of LLMs as assessors to automate the TTCW evaluation, revealing that none of the LLMs positively correlate with the expert assessments.