mlsanityData to 7 Oct 2026

A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writing

arXiv 2310.08433 ↗First posted 12 Oct 2023cs.CL cs.CYcited by 98

Not on the map. Why?

Abstract (arXiv)

We evaluate a range of recent LLMs on English creative writing, a challenging and complex task that requires imagination, coherence, and style. We use a difficult, open-ended scenario chosen to avoid training data reuse: an epic narration of a single combat between Ignatius J. Reilly, the protagonist of the Pulitzer Prize-winning novel A Confederacy of Dunces (1980), and a pterodactyl, a prehistoric flying reptile. We ask several LLMs and humans to write such a story and conduct a human evalution involving various criteria such as fluency, coherence, originality, humor, and style. Our results show that some state-of-the-art commercial LLMs match or slightly outperform our writers in most dimensions; whereas open-source LLMs lag behind. Humans retain an edge in creativity, while humor shows a binary divide between LLMs that can handle it comparably to humans and those that fail at it. We discuss the implications and limitations of our study and suggest directions for future research.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 31 of its 70 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

31 of 70 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2023A Survey of Large Language Modelscited by 2,537
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
19 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  3. 2022A Survey on In-context Learningcited by 717
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  5. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  6. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  7. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  8. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  9. 2020A Knowledge-Enhanced Pretraining Model for Commonsense Story Generationcited by 116
  10. 2021The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generationcited by 96
  11. 2019Do Massively Pretrained Language Models Make Better Storytellers?cited by 90
  12. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  13. 2020MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Modelscited by 74
  14. 2023On the Creativity of Large Language Modelscited by 57
  15. 2021Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherencecited by 45
  16. 2023ChatGPT is fun, but it is not funny! Humor is still challenging Large Language Modelscited by 23
  17. 2023ChatGPT: More than a Weapon of Mass Deception, Ethical challenges and responses from the Human-Centered Artificial Intelligence (HCAI) perspectivecited by 22
  18. 2023Bits of Grass: Does GPT already know how to write like Whitman?cited by 13
  19. 2023Simulating H.P. Lovecraft horror literature with the ChatGPT large language modelcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 35; 2025: 48; 2026: 14.

98 papers.

Sort:

By area:

  1. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 95
  2. 2024MarkLLM: An Open-Source Toolkit for LLM Watermarkingcited by 80
  3. 2024V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimizationcited by 67
  4. 2024LLM Discussion: Enhancing the Creativity of Large Language Models via Discussion Framework and Role-Playcited by 65
  5. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  6. 2025Base Models Beat Aligned Models at Randomness and Creativitycited by 48
  7. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 43
  8. 2024MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Timecited by 40
  9. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 38
  10. 2024Divergent Creativity in Humans and Large Language Modelscited by 29
  11. 2025A Survey on Parallel Text Generation: From Parallel Decoding to Diffusion Language Modelscited by 27
  12. 2023Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytellingcited by 26
38 more
  1. 2024Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?cited by 26
  2. 2024Multi-Agent Large Language Models for Conversational Task-Solvingcited by 24
  3. 2024T-RAG: Lessons from the LLM Trenchescited by 23
  4. 2024Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Taskscited by 22
  5. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 22
  6. 2025DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMscited by 21
  7. 2024Personalized Text Generation with Fine-Grained Linguistic Controlcited by 20
  8. 2024Revealing the Barriers of Language Agents in Planningcited by 20
  9. 2024ControlAgent: Automating Control System Design via Novel Integration of LLM Agents and Domain Expertisecited by 20
  10. 2025Automated Creativity Evaluation for Large Language Models: A Reference-Based Approachcited by 20
  11. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  12. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  13. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  14. 2025Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLMcited by 14
  15. 2024Enhancing Creativity in Large Language Models through Associative Thinking Strategiescited by 13
  16. 2024From Prompt Engineering to Prompt Science With Human in the Loopcited by 12
  17. 2025BookWorld: From Novels to Interactive Agent Societies for Creative Story Generationcited by 12
  18. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 12
  19. 2025RECAST: Expanding the Boundaries of LLMs' Complex Instruction Following with Multi-Constraint Datacited by 11
  20. 2025Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewardscited by 11
  21. 2024Graph-based Confidence Calibration for Large Language Modelscited by 10
  22. 2025Generalizability of Large Language Model-Based Agents: A Comprehensive Surveycited by 10
  23. 2025BILLY: Steering Large Language Models via Merging Persona Vectors for Creative Generationcited by 10
  24. 2024Measuring Psychological Depth in Language Modelscited by 9
  25. 2024Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problemscited by 9
  26. 2025Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Modelscited by 9
  27. 2024MirrorStories: Reflecting Diversity through Personalized Narrative Generation with Large Language Modelscited by 8
  28. 2025Calibrating Uncertainty Quantification of Multi-Modal LLMs using Groundingcited by 8
  29. 2025IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignmentcited by 8
  30. 2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
  31. 2024Creativity in AI: Progresses and Challengescited by 7
  32. 2025The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writingcited by 7
  33. 2025KatFishNet: Detecting LLM-Generated Korean Text through Linguistic Feature Analysiscited by 6
  34. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  35. 2024Emerging Opportunities of Using Large Language Models for Translation Between Drug Molecules and Indicationscited by 5
  36. 2024Is persona enough for personality? Using ChatGPT to reconstruct an agent's latent personality from simple descriptionscited by 5
  37. 2024Do LLMs Agree on the Creativity Evaluation of Alternative Uses?cited by 5
  38. 2025Generative AI and Creativity: A Systematic Literature Review and Meta-Analysiscited by 5

50 most cited of 98 All 98 (JSON)