# Evaluating LLM Story Generation through Large-scale Network Analysis of Social Structures

arXiv: https://arxiv.org/abs/2510.18932 · HTML: https://mlsanity.com/paper/2510.18932/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 27 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 8 of 8)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Are Large Language Models Capable of Generating Human-Level Narratives?](https://mlsanity.com/paper/2407.13248/) (2024)
- [The Next Chapter: A Study of Large Language Models in Storytelling](https://mlsanity.com/paper/2301.09790/) (2023)
- [Evaluating Creative Short Story Generation in Humans and Large Language Models](https://mlsanity.com/paper/2411.02316/) (2024)

## Built on by (most cited first, 2 of 2)

- [StoryScope: Investigating idiosyncrasies in AI fiction](https://mlsanity.com/paper/2604.03136/) (2026)
- [Multilingual in Name Only? Cultural and Linguistic Weaknesses of LLMs in Urdu](https://mlsanity.com/paper/2609.10758/) (2026)

## Abstract (arXiv)

Evaluating the creative capabilities of large language models (LLMs) in complex tasks often requires human assessments that are difficult to scale. We introduce a novel, scalable methodology for evaluating LLM story generation by analyzing underlying social structures in narratives as signed character networks. To demonstrate its effectiveness, we conduct a large-scale comparative analysis using networks from over 1,200 stories, generated by four leading LLMs (GPT-4o, GPT-4o mini, Gemini 1.5 Pro, and Gemini 1.5 Flash) and a human-written corpus. Our findings, based on network properties like density, clustering, and signed edge weights, show that LLM-generated stories consistently exhibit a strong bias toward tightly-knit, positive relationships, which aligns with findings from prior research using human assessment. Our proposed approach provides a valuable tool for evaluating limitations and tendencies in the creative storytelling of current and future LLMs.