# Small Language Models can Outperform Humans in Short Creative Writing: A Study Comparing SLMs with Humans and LLMs

arXiv: https://arxiv.org/abs/2409.11547 · HTML: https://mlsanity.com/paper/2409.11547/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Is Temperature the Creativity Parameter of Large Language Models?](https://mlsanity.com/paper/2405.00492/) (2024)
- [Do Massively Pretrained Language Models Make Better Storytellers?](https://mlsanity.com/paper/1909.10705/) (2019)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [Weaver: Foundation Models for Creative Writing](https://mlsanity.com/paper/2401.17268/) (2024)
- [Pron vs Prompt: Can Large Language Models already Challenge a World-Class Fiction Author at Creative Text Writing?](https://mlsanity.com/paper/2407.01119/) (2024)
- [Small But Funny: A Feedback-Driven Approach to Humor Distillation](https://mlsanity.com/paper/2402.18113/) (2024)
- [MPST: A Corpus of Movie Plot Synopses with Tags](https://mlsanity.com/paper/1802.07858/) (2018)

## Built on by (most cited first, 17 of 17)

- [Diverse Preference Optimization](https://mlsanity.com/paper/2501.18101/) (2025)
- [Evaluating Creative Short Story Generation in Humans and Large Language Models](https://mlsanity.com/paper/2411.02316/) (2024)
- [The Reader is the Metric: How Textual Features and Reader Profiles Explain Conflicting Evaluations of AI Creative Writing](https://mlsanity.com/paper/2506.03310/) (2025)
- [Pencils to Pixels: A Systematic Study of Creative Drawings across Children, Adults and AI](https://mlsanity.com/paper/2502.05999/) (2025)
- [Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variability](https://mlsanity.com/paper/2504.12320/) (2025)
- [First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation](https://mlsanity.com/paper/2511.04715/) (2025)
- [ArgCMV: An Argument Summarization Benchmark for the LLM-era](https://mlsanity.com/paper/2508.19580/) (2025)
- [Beyond Divergent Creativity: A Human-Based Evaluation of Creativity in Large Language Models](https://mlsanity.com/paper/2601.20546/) (2026)
- [Investigating Gender Bias in LLM-Generated Stories via Psychological Stereotypes](https://mlsanity.com/paper/2508.03292/) (2025)
- [AI Fiction in the Wild](https://mlsanity.com/paper/2606.22748/) (2026)
- [Casting a SPELL: Sentence Pairing Exploration for LLM Limitation-breaking](https://mlsanity.com/paper/2512.21236/) (2025)
- [On the Creativity of AI Agents](https://mlsanity.com/paper/2604.13242/) (2026)
- [Beyond Reproduction: A Paired-Task Framework for Assessing LLM Comprehension and Creativity in Literary Translation](https://mlsanity.com/paper/2604.18169/) (2026)
- [Do Large Language Models Always Tell The Same Stories?](https://mlsanity.com/paper/2606.17350/) (2026)
- [High-quality generation of dynamic game content via small language models: A proof of concept](https://mlsanity.com/paper/2601.23206/) (2026)
- [Constrained Semantic Decompression in LLMs through Persian Proverb-Conditioned Story Generation](https://mlsanity.com/paper/2606.12599/) (2026)
- [Synthetic Consumer Insight Generation with Large Language Models](https://mlsanity.com/paper/2607.05761/) (2026)

## Abstract (arXiv)

In this paper, we evaluate the creative fiction writing abilities of a fine-tuned small language model (SLM), BART-large, and compare its performance to human writers and two large language models (LLMs): GPT-3.5 and GPT-4o. Our evaluation consists of two experiments: (i) a human study in which 68 participants rated short stories from humans and the SLM on grammaticality, relevance, creativity, and attractiveness, and (ii) a qualitative linguistic analysis examining the textual characteristics of stories produced by each model. In the first experiment, BART-large outscored average human writers overall (2.11 vs. 1.85), a 14% relative improvement, though the slight human advantage in creativity was not statistically significant. In the second experiment, qualitative analysis showed that while GPT-4o demonstrated near-perfect coherence and used less cliche phrases, it tended to produce more predictable language, with only 3% of its synopses featuring surprising associations (compared to 15% for BART). These findings highlight how model size and fine-tuning influence the balance between creativity, fluency, and coherence in creative writing tasks, and demonstrate that smaller models can, in certain contexts, rival both humans and larger models.