mlsanityData to 14 Oct 2026

Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors

arXiv 2205.12854 ↗First posted 25 May 2022cs.CL cs.AIcited by 104

Not on the map. Why?

Abstract (arXiv)

The propensity of abstractive summarization models to make factual errors has been studied extensively, including design of metrics to detect factual errors and annotation of errors in current systems' outputs. However, the ever-evolving nature of summarization systems, metrics, and annotated benchmarks makes factuality evaluation a moving target, and drawing clear comparisons among metrics has become increasingly difficult. In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models. Critically, our analysis shows that much of the recent improvement in the factuality detection space has been on summaries from older (pre-Transformer) models instead of more relevant recent summarization models. We further perform a finer-grained analysis per error-type and find similar performance variance across error types for different factuality metrics. Our results show that no one metric is superior in all settings or for all error types, and we provide recommendations for best practices given these insights.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Builds on 43 of its 64 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

43 of 64 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  9. 2015Teaching Machines to Read and Comprehendcited by 1,049
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  12. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
31 more
  1. 2019Text Summarization with Pretrained Encoderscited by 556
  2. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  3. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  4. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  5. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  6. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 246
  7. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  8. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  9. 2018Bottom-Up Abstractive Summarizationcited by 177
  10. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  11. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  12. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  13. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  14. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  15. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  16. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  17. 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
  18. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 76
  19. 2018Multi-Reward Reinforced Summarization with Saliency and Entailmentcited by 73
  20. 2020What Have We Achieved on Text Summarization?cited by 65
  21. 2018Learning to Extract Coherent Summary via Deep Reinforcement Learningcited by 64
  22. 2018Neural Document Summarization by Jointly Learning to Score and Select Sentencescited by 56
  23. 2021Improving Factual Consistency of Abstractive Summarization via Question Answeringcited by 55
  24. 2018A Unified Model for Extractive and Abstractive Summarization using Inconsistency Losscited by 47
  25. 2018BanditSum: Extractive Summarization as a Contextual Banditcited by 46
  26. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44
  27. 2021ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Miningcited by 38
  28. 2018Improving Abstraction in Text Summarizationcited by 35
  29. 2018Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generationcited by 25
  30. 2021EmailSum: Abstractive Email Thread Summarizationcited by 24
  31. 2018Closed-Book Training to Improve Summarization Encoder Memorycited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 24; 2024: 37; 2025: 20; 2026: 14.

104 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  3. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  4. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  5. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  6. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 108
  7. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  8. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 77
  9. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  10. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  11. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  12. 2024Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scorescited by 50
38 more
  1. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 49
  2. 2023Effective Large Language Model Adaptation for Improved Grounding and Citation Generationcited by 45
  3. 2022Prompted Opinion Summarization with GPT-3.5cited by 44
  4. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  5. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  6. 2023LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyondcited by 38
  7. 2024FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extractioncited by 28
  8. 2023It Takes Two to Tango: Navigating Conceptualizations of NLP Tasks and Measurements of Performancecited by 26
  9. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 26
  10. 2024Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trendscited by 25
  11. 2023Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLMscited by 24
  12. 2024FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMscited by 24
  13. 2024Fine-Grained Natural Language Inference Based Faithfulness Evaluation for Diverse Summarisation Taskscited by 23
  14. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
  15. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboardscited by 20
  16. 2024Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writerscited by 19
  17. 2024FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidencecited by 18
  18. 2024From Single to Multi: How LLMs Hallucinate in Multi-Document Summarizationcited by 17
  19. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  20. 2025GINGER: Grounded Information Nugget-Based Generation of Responsescited by 16
  21. 2024FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Documentcited by 15
  22. 2023UMASS_BioNLP at MEDIQA-Chat 2023: Can LLMs generate high-quality synthetic note-oriented doctor-patient conversations?cited by 14
  23. 2024Localizing Factual Inconsistencies in Attributable Text Generationcited by 14
  24. 2025FactCG: Enhancing Fact Checkers with Graph-Based Multi-Hop Datacited by 14
  25. 2025Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifierscited by 13
  26. 2022Hierarchical3D Adapters for Long Video-to-text Summarizationcited by 12
  27. 2022Analyzing and Evaluating Faithfulness in Dialogue Summarizationcited by 12
  28. 2025GraphCheck: Breaking Long-Term Text Barriers with Extracted Knowledge Graph-Powered Fact-Checkingcited by 12
  29. 2023AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generationcited by 11
  30. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 11
  31. 2025Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languagescited by 11
  32. 2022Improved Beam Search for Hallucination Mitigation in Abstractive Summarizationcited by 10
  33. 2023Evaluating Factual Consistency of Summaries with Large Language Modelscited by 10
  34. 2023Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarizationcited by 10
  35. 2024Evaluating the Factuality of Zero-shot Summarizers Across Varied Domainscited by 10
  36. 2024Face4RAG: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinesecited by 10
  37. 2025Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluationcited by 10
  38. 2024STORYSUMM: Evaluating Faithfulness in Story Summarizationcited by 9

50 most cited of 104 All 104 (JSON)