mlsanityData to 7 Oct 2026

Art or Artifice? Large Language Models and the False Promise of Creativity

arXiv 2309.14556 ↗First posted 25 Sep 2023cs.CL cs.AI cs.HCcited by 168

Not on the map. Why?

Abstract (arXiv)

Researchers have argued that large language models (LLMs) exhibit high-quality writing capabilities from blogs to stories. However, evaluating objectively the creativity of a piece of writing is challenging. Inspired by the Torrance Test of Creative Thinking (TTCT), which measures creativity as a process, we use the Consensual Assessment Technique [3] and propose the Torrance Test of Creative Writing (TTCW) to evaluate creativity as a product. TTCW consists of 14 binary tests organized into the original dimensions of Fluency, Flexibility, Originality, and Elaboration. We recruit 10 creative writers and implement a human assessment of 48 stories written either by professional authors or LLMs using TTCW. Our analysis shows that LLM-generated stories pass 3-10X less TTCW tests than stories written by professionals. In addition, we explore the use of LLMs as assessors to automate the TTCW evaluation, revealing that none of the LLMs positively correlate with the expert assessments.

Key facts. Cited by 168 papers in the map (data through 7 Oct 2026). Builds on 23 of its 83 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

23 of 83 references.

References · 23

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2020Learning to summarize from human feedbackcited by 2,562
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  6. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  7. 2023How is ChatGPT's behavior changing over time?cited by 265
  8. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  9. 2018Plan-And-Write: Towards Better Automatic Storytellingcited by 232
  10. 2022Re3: Generating Longer Stories With Recursive Reprompting and Revisioncited by 193
  11. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  12. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
11 more
  1. 2020PlotMachines: Outline-Conditioned Generation with Dynamic Plot State Trackingcited by 104
  2. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  3. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  4. 2020Content Planning for Neural Story Generation with Aristotelian Rescoringcited by 92
  5. 2023Large Language Models Enable Few-Shot Clusteringcited by 72
  6. 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
  7. 2023LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyondcited by 37
  8. 2023Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writerscited by 32
  9. 2020Collaborative Storytelling with Large-scale Neural Language Modelscited by 21
  10. 2023Next Steps for Human-Centered Generative AI: A Technical Perspectivecited by 13
  11. 2023Ambient Adventures: Teaching ChatGPT on Developing Complex Storiescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 48; 2025: 68; 2026: 49.

168 papers.

Sort:

By area:

  1. 2025LLMs Get Lost In Multi-Turn Conversationcited by 400
  2. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  3. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 98
  4. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
  5. 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 78
  6. 2024Safeguarding Large Language Models: A Surveycited by 73
  7. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 65
  8. 2024Building Guardrails for Large Language Modelscited by 61
  9. 2024Agents' Room: Narrative Generation through Multi-step Collaborationcited by 57
  10. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  11. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  12. 2024Echoes in AI: Quantifying lack of plot diversity in LLM outputscited by 47
38 more
  1. 2024AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Textcited by 40
  2. 2025Exploring the Impact of Temperature on Large Language Models:Hot or Cold?cited by 40
  3. 2025People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated textcited by 39
  4. 2025LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writingcited by 39
  5. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 38
  6. 2024Assessing and Understanding Creativity in Large Language Modelscited by 35
  7. 2026DeepResearch Bench II: Diagnosing Deep Research Agents via Rubrics from Expert Reportscited by 33
  8. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  9. 2024Aligning Language Models with Demonstrated Feedbackcited by 32
  10. 2024Divergent Creativity in Humans and Large Language Modelscited by 29
  11. 2024A Robot Walks into a Bar: Can Language Models Serve as Creativity Support Tools for Comedy? An Evaluation of LLMs' Humour Alignment with Comedianscited by 29
  12. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  13. 2024Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?cited by 26
  14. 2024Evaluating and Enhancing Large Language Models for Novelty Assessment in Scholarly Publicationscited by 26
  15. 2025LLM Probability Concentration: How Alignment Shrinks the Generative Horizoncited by 26
  16. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 24
  17. 2025Creative Preference Optimizationcited by 24
  18. 2024Patchview: LLM-Powered Worldbuilding with Generative Dust and Magnet Visualizationcited by 23
  19. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 22
  20. 2024Evaluating Creative Short Story Generation in Humans and Large Language Modelscited by 21
  21. 2025Automated Creativity Evaluation for Large Language Models: A Reference-Based Approachcited by 20
  22. 2024Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writerscited by 19
  23. 2024Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMscited by 17
  24. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  25. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  26. 2024Navigating the Path of Writing: Outline-guided Text Generation with Large Language Modelscited by 15
  27. 2024Eliciting Informative Text Evaluations with Large Language Modelscited by 14
  28. 2026The Need for a Socially-Grounded Persona Framework for User Simulationcited by 14
  29. 2024Enhancing Creativity in Large Language Models through Associative Thinking Strategiescited by 13
  30. 2024MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generationcited by 13
  31. 2025QUDsim: Quantifying Discourse Similarities in LLM-Generated Textcited by 13
  32. 2023Aspects of human memory and Large Language Modelscited by 12
  33. 2025EvalAgent: Discovering Implicit Evaluation Criteria from the Webcited by 11
  34. 2025Aligning LLMs by Predicting Preferences from User Writing Samplescited by 11
  35. 2025Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewardscited by 11
  36. 2024Subtle Biases Need Subtler Measures: Dual Metrics for Evaluating Representative and Affinity Bias in Large Language Modelscited by 10
  37. 2025LiTransProQA: an LLM-based Literary Translation evaluation metric with Professional Question Answeringcited by 10
  38. 2025Help Me Write a Story: Evaluating LLMs' Ability to Generate Writing Feedbackcited by 10

50 most cited of 168 All 168 (JSON)