mlsanityData to 14 Oct 2026

Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writers

arXiv 2403.01061 ↗First posted 2 Mar 2024cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

We evaluate recent Large Language Models (LLMs) on the challenging task of summarizing short stories, which can be lengthy, and include nuanced subtext or scrambled timelines. Importantly, we work directly with authors to ensure that the stories have not been shared online (and therefore are unseen by the models), and to obtain informed evaluations of summary quality using judgments from the authors themselves. Through quantitative and qualitative analysis grounded in narrative theory, we compare GPT-4, Claude-2.1, and LLama-2-70B. We find that all three models make faithfulness mistakes in over 50% of summaries and struggle with specificity and interpretation of difficult subtext. We additionally demonstrate that LLM ratings and other automatic metrics for summary quality do not correlate well with the quality ratings from the writers.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 37 of its 62 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

37 of 62 references.

References · 37

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  4. 2015Teaching Machines to Read and Comprehendcited by 1,049
  5. 2023Large Language Modelscited by 780
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  7. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  8. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  9. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 294
  10. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  11. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  12. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
25 more
  1. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  2. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  3. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  4. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 172
  5. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  6. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  7. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  8. 2022Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehensioncited by 80
  9. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 79
  10. 2022Help me write a poem: Instruction Tuning as a Vehicle for Collaborative Poetry Writingcited by 71
  11. 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
  12. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  13. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  14. 2024OpenToM: A Comprehensive Benchmark for Evaluating Theory-of-Mind Reasoning Capabilities of Large Language Modelscited by 54
  15. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  16. 2023Summarization is (Almost) Deadcited by 44
  17. 2023Evaluation of African American Language Bias in Natural Language Generationcited by 36
  18. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  19. 2023Creativity Support in the Age of Large Language Models: An Empirical Study Involving Emerging Writerscited by 32
  20. 2020Exploring Content Selection in Summarization of Novel Chapterscited by 29
  21. 2023Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytellingcited by 27
  22. 2022SNaC: Coherence Error Detection for Narrative Summarizationcited by 19
  23. 2024GhostWriter: Augmenting Collaborative Human-AI Writing Experiences Through Personalization and Agencycited by 18
  24. 2023Inspo: Writing with Crowds Alongside AIcited by 5
  25. 2023MUG: A General Meeting Understanding and Generation Benchmarkcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 6.

19 papers.

Sort:

By area:

  1. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  2. 2024HEART-felt Narratives: Tracing Empathy and Narrative Style in Personal Stories with LLMscited by 16
  3. 2025GINGER: Grounded Information Nugget-Based Generation of Responsescited by 16
  4. 2024STORYSUMM: Evaluating Faithfulness in Story Summarizationcited by 9
  5. 2024Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playingcited by 9
  6. 2025LLMs as Span Annotators: A Comparative Study of LLMs and Humanscited by 7
  7. 2024Empowering Meta-Analysis: Leveraging Large Language Models for Scientific Synthesiscited by 4
  8. 2026Using LLM-as-a-Judge/Jury to Advance Scalable, Clinically-Validated Safety Evaluations of Model Responses to Users Demonstrating Psychosiscited by 3
  9. 2025Guiding Generative Storytelling with Knowledge Graphscited by 2
  10. 2025Literary Evidence Retrieval via Long-Context Language Modelscited by 1
  11. 2025UiS-IAI@LiveRAG: Retrieval-Augmented Information Nugget-Based Generation of Responsescited by 1
  12. 2025Are You There God? Lightweight Narrative Annotation of Christian Fiction with LMscited by 1
7 more
  1. 2026NARRA-Gym for Evaluating Interactive Narrative Agentscited by 1
  2. 2025Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understandingcited by 0
  3. 2025PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMscited by 0
  4. 2026LitVISTA: A Benchmark for Narrative Orchestration in Literary Textcited by 0
  5. 2026Beneath the Surface: Investigating LLMs' Capabilities for Communicating with Subtextcited by 0
  6. 2026FocalLens: Visualizing Narratives through Focalizationcited by 0
  7. 2026Evaluating Epistemic Guardrails in AI Reading Assistants: A Behavioral Audit of a Minimal Prototypecited by 0