mlsanityData to 14 Oct 2026

NarraBench: A Comprehensive Framework for Narrative Benchmarking

arXiv 2510.09869 ↗First posted 10 Oct 2025cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

We present NarraBench, a theory-informed taxonomy of narrative-understanding tasks, as well as an associated survey of 78 existing benchmarks in the area. We find significant need for new evaluations covering aspects of narrative understanding that are either overlooked in current work or are poorly aligned with existing metrics. Specifically, we estimate that only 27% of narrative tasks are well captured by existing benchmarks, and we note that some areas -- including narrative events, style, perspective, and revelation -- are nearly absent from current evaluations. We also note the need for increased development of benchmarks capable of assessing constitutively subjective and perspectival aspects of narrative, that is, aspects for which there is generally no single correct answer. Our taxonomy, survey, and methodology are of value to NLP researchers seeking to test LLM narrative understanding.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 45 of its 104 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

45 of 104 references.

References · 45

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  5. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  6. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  7. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  8. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  9. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  10. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  11. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  12. 2024Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoningcited by 88
33 more
  1. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  2. 2024Are Large Language Models Capable of Generating Human-Level Narratives?cited by 79
  3. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 74
  4. 2024ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMcited by 73
  5. 2016The emotional arcs of stories are dominated by six basic shapescited by 71
  6. 2024Artifacts or Abduction: How Do LLMs Answer Multiple-Choice Questions Without the Question?cited by 71
  7. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
  8. 2022MAVEN-ERE: A Unified Large-scale Dataset for Event Coreference, Temporal, Causal, and Subevent Relation Extractioncited by 66
  9. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  10. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 33
  11. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 32
  12. 2023DialogBench: Evaluating LLMs as Human-like Dialogue Systemscited by 31
  13. 2021Goal-Oriented Script Constructioncited by 30
  14. 2024RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Contentcited by 26
  15. 2025CoSER: A Comprehensive Literary Dataset and Framework for Training and Evaluating LLM Role-Playing and Persona Simulationcited by 26
  16. 2025Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detectioncited by 24
  17. 2023Where Do People Tell Stories Online? Story Detection Across Online Communitiescited by 17
  18. 2020An automated pipeline for the discovery of conspiracy and conspiracy theory narrative frameworks: Bridgegate, Pizzagate and storytelling on the webcited by 16
  19. 2020Measuring Information Propagation in Literary Social Networkscited by 15
  20. 2021From Discourse to Narrative: Knowledge Projection for Event Relation Extractioncited by 14
  21. 2025PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluationcited by 14
  22. 2023Human-in-the-Loop Schema Inductioncited by 13
  23. 2024Time Awareness in Large Language Models: Benchmarking Fact Recall Across Timecited by 13
  24. 2022VeriDark: A Large-Scale Benchmark for Authorship Verification on the Dark Webcited by 10
  25. 2024Abstention vs. Hallucination: Benchmarking LLM Source Attribution for Scientific Citationscited by 10
  26. 2024Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMscited by 10
  27. 2023Are NLP Models Good at Tracing Thoughts: An Overview of Narrative Understandingcited by 9
  28. 2024STORYSUMM: Evaluating Faithfulness in Story Summarizationcited by 9
  29. 2025MORABLES: A Benchmark for Assessing Abstract Moral Reasoning in LLMs with Fablescited by 9
  30. 2022NarraSum: A Large-Scale Dataset for Abstractive Narrative Summarizationcited by 5
  31. 2025The Zero Body Problem: Probing LLM Use of Sensory Languagecited by 5
  32. 2021SentimentArcs: A Novel Method for Self-Supervised Sentiment Analysis of Time Series Shows SOTA Transformers Can Struggle Finding Narrative Arcscited by 4
  33. 2024Evaluating Computational Representations of Character: An Austen Character Similarity Benchmarkcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026StoryScope: Investigating idiosyncrasies in AI fictioncited by 9
  2. 2026AI as Entertainmentcited by 2
  3. 2026Lessons Without Borders? Evaluating Cultural Alignment of LLMs Using Multilingual Story Moral Generationcited by 2
  4. 2026AI as a Tool for Simulation-Based Experiments in Literary Studiescited by 2
  5. 2026Characterizing Narrative Content in Web-scale LLM Pretraining Datacited by 2
  6. 2026From Personas to Plot: Character-Grounded Multi-Agent Story Generation for Long-Form Narrativescited by 2
  7. 2026GraphLit: Learning Text-Enriched Dynamic Character Network Representations for Literary Studycited by 1
  8. 2026Evaluating Multimodal Narrative Understanding of Popular Hollywood Filmscited by 1
  9. 2026When Models Know More Than They Say: Probing Analogical Reasoning in LLMscited by 0
  10. 2026SAGE: Hierarchical LLM-Based Literary Evaluation through Ontology-Grounded Interpretive Dimensionscited by 0
  11. 2026GENIE: A Fine-Grained Measure for Noveltycited by 0
  12. 2026SAGE: A Hierarchical Framework for Evaluating Interpretive Literary Quality in Narrativescited by 0
1 more
  1. 2026SlopShape: Identifying AI-Generated Commercial Web Contentcited by 0