mlsanityData to 7 Oct 2026

STORYSUMM: Evaluating Faithfulness in Story Summarization

arXiv 2407.06501 ↗First posted 9 Jul 2024cs.AI cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Human evaluation has been the gold standard for checking faithfulness in abstractive summarization. However, with a challenging source domain like narrative, multiple annotators can agree a summary is faithful, while missing details that are obvious errors only once pointed out. We therefore introduce a new dataset, STORYSUMM, comprising LLM summaries of short stories with localized faithfulness labels and error explanations. This benchmark is for evaluation methods, testing whether a given method can detect challenging inconsistencies. Using this dataset, we first show that any one human annotation protocol is likely to miss inconsistencies, and we advocate for pursuing a range of methods when establishing ground truth for a summarization dataset. We finally test recent automatic metrics and find that none of them achieve more than 70% balanced accuracy on this task, demonstrating that it is a challenging benchmark for future work in faithfulness evaluation.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 9 Jul 2024.

Builds on

25 of 37 references.

References · 25

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  4. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  5. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  6. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  7. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  8. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  9. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  10. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  11. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  12. 2019Neural Text Summarization: A Critical Evaluationcited by 189
13 more
  1. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  2. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  3. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  4. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  5. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 75
  6. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  7. 2020What Have We Achieved on Text Summarization?cited by 65
  8. 2024FABLES: Evaluating faithfulness and content selection in book-length summarizationcited by 65
  9. 2023Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrongcited by 51
  10. 2024Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debatecited by 48
  11. 2022Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarizationcited by 34
  12. 2022SNaC: Coherence Error Detection for Narrative Summarizationcited by 19
  13. 2024Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writerscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 2.

9 papers.

  1. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 13
  2. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  3. 2025AraHalluEval: A Fine-grained Hallucination Evaluation Framework for Arabic LLMscited by 12
  4. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
  5. 2025Agent-as-Judge for Factual Summarization of Long Narrativescited by 9
  6. 2025Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understandingcited by 0
  7. 2025Faithfulness metric fusion: Improving the evaluation of LLM trustworthiness across domainscited by 0
  8. 2026NARRA-Gym for Evaluating Interactive Narrative Agentscited by 0
  9. 2026Emergent Unfaithfulness: How Alignment Training Causes Language Models to Silently Override Task Faithfulnesscited by 0