# Echoes in AI: Quantifying lack of plot diversity in LLM outputs

arXiv: https://arxiv.org/abs/2501.00273 · HTML: https://mlsanity.com/paper/2501.00273/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 31 Dec 2024.

## Builds on (most cited first, 16 of 16)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers](https://mlsanity.com/paper/2409.04109/) (2024)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Generative artificial intelligence enhances creativity but reduces the diversity of novel content](https://mlsanity.com/paper/2312.00506/) (2023)
- [Homogenization Effects of Large Language Models on Human Creative Ideation](https://mlsanity.com/paper/2402.01536/) (2024)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [Is Temperature the Creativity Parameter of Large Language Models?](https://mlsanity.com/paper/2405.00492/) (2024)
- [Delving into LLM-assisted writing in biomedical publications through excess vocabulary](https://mlsanity.com/paper/2406.07016/) (2024)
- [Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores](https://mlsanity.com/paper/2403.00553/) (2024)
- [Creativity Has Left the Chat: The Price of Debiasing Language Models](https://mlsanity.com/paper/2406.05587/) (2024)
- [Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling](https://mlsanity.com/paper/2310.12902/) (2023)
- [Guiding and Diversifying LLM-Based Story Generation via Answer Set Programming](https://mlsanity.com/paper/2406.00554/) (2024)

## Built on by (most cited first, 30 of 48)

- [Correlated Errors in Large Language Models](https://mlsanity.com/paper/2506.07962/) (2025)
- [Modifying Large Language Model Post-Training for Diverse Creative Writing](https://mlsanity.com/paper/2503.17126/) (2025)
- [The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Models](https://mlsanity.com/paper/2502.11266/) (2025)
- [Creative Preference Optimization](https://mlsanity.com/paper/2505.14442/) (2025)
- [Post-training Large Language Models for Diverse High-Quality Responses](https://mlsanity.com/paper/2509.04784/) (2025)
- [What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models](https://mlsanity.com/paper/2510.04226/) (2025)
- [QUDsim: Quantifying Discourse Similarities in LLM-Generated Text](https://mlsanity.com/paper/2504.09373/) (2025)
- [TALES: A Taxonomy and Analysis of Cultural Representations in LLM-generated Stories](https://mlsanity.com/paper/2511.21322/) (2025)
- [Diverse, not Short: A Length-Controlled Data Selection Strategy for Improving Response Diversity of Language Models](https://mlsanity.com/paper/2505.16245/) (2025)
- [StoryScope: Investigating idiosyncrasies in AI fiction](https://mlsanity.com/paper/2604.03136/) (2026)
- [Readers Prefer Outputs of AI Trained on Copyrighted Books over Expert Human Writers](https://mlsanity.com/paper/2510.13939/) (2025)
- [Optimizing Diversity and Quality through Base-Aligned Model Collaboration](https://mlsanity.com/paper/2511.05650/) (2025)
- [CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models](https://mlsanity.com/paper/2512.07890/) (2025)
- [Multi-Novelty: Improve the Diversity and Novelty of Contents Generated by Large Language Models via inference-time Multi-Views Brainstorming](https://mlsanity.com/paper/2502.12700/) (2025)
- [LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writers](https://mlsanity.com/paper/2602.16162/) (2026)
- [Research Borderlands: Analysing Writing Across Research Cultures](https://mlsanity.com/paper/2506.00784/) (2025)
- [Addressing LLM Diversity by Infusing Random Concepts](https://mlsanity.com/paper/2601.18053/) (2026)
- [Markovian Generation Chains in Large Language Models](https://mlsanity.com/paper/2603.11228/) (2026)
- [Style over Story: Measuring LLM Narrative Preferences via Structured Selection](https://mlsanity.com/paper/2510.02025/) (2025)
- [Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Tasks](https://mlsanity.com/paper/2512.01725/) (2025)
- [Evolutionary Reinforcement Learning based AI tutor for Socratic Interdisciplinary Instruction](https://mlsanity.com/paper/2512.11930/) (2025)
- [Epistemic diversity across language models mitigates knowledge collapse](https://mlsanity.com/paper/2512.15011/) (2025)
- [The AI Fiction Paradox](https://mlsanity.com/paper/2603.13545/) (2026)
- [Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs](https://mlsanity.com/paper/2605.11128/) (2026)
- [AI as a Tool for Simulation-Based Experiments in Literary Studies](https://mlsanity.com/paper/2606.02293/) (2026)
- [GFlowRL: Scaling Distribution-Matching RL to Large Language Models](https://mlsanity.com/paper/2607.13394/) (2026)
- [Are LLMs becoming similarly creative? Evidence from three years of models](https://mlsanity.com/paper/2608.19437/) (2026)
- [Too Open for Opinion? Embracing Open-Endedness in Large Language Models for Social Simulation](https://mlsanity.com/paper/2510.13884/) (2025)
- [TeamLLM: A Human-Like Team-Oriented Collaboration Framework for Multi-Step Contextualized Tasks](https://mlsanity.com/paper/2604.06765/) (2026)
- [Spoiler Alert: Narrative Forecasting as a Metric for Tension in LLM Storytelling](https://mlsanity.com/paper/2604.09854/) (2026)

## Abstract (arXiv)

With rapid advances in large language models (LLMs), there has been an increasing application of LLMs in creative content ideation and generation. A critical question emerges: can current LLMs provide ideas that are diverse enough to truly bolster collective creativity? We examine two state-of-the-art LLMs, GPT-4 and LLaMA-3, on story generation and discover that LLM-generated stories often consist of plot elements that are echoed across a number of generations. To quantify this phenomenon, we introduce the Sui Generis score, an automatic metric that measures the uniqueness of a plot element among alternative storylines generated using the same prompt under an LLM. Evaluating on 100 short stories, we find that LLM-generated stories often contain combinations of idiosyncratic plot elements echoed frequently across generations and across different LLMs, while plots from the original human-written stories are rarely recreated or even echoed in pieces. Moreover, our human evaluation shows that the ranking of Sui Generis scores among story segments correlates moderately with human judgment of surprise level, even though score computation is completely automatic without relying on human judgment.