mlsanityData to 7 Oct 2026

Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

arXiv 2104.13346 ↗First posted 27 Apr 2021cs.CLcited by 285

Not on the map. Why?

Abstract (arXiv)

Modern summarization models generate highly fluent but often factually unreliable outputs. This motivated a surge of metrics attempting to measure the factuality of automatically generated summaries. Due to the lack of common benchmarks, these metrics cannot be compared. Moreover, all these methods treat factuality as a binary concept and fail to provide deeper insights into the kinds of inconsistencies made by different systems. To address these limitations, we devise a typology of factual errors and use it to collect human annotations of generated summaries from state-of-the-art summarization systems for the CNN/DM and XSum datasets. Through these annotations, we identify the proportion of different categories of factual errors in various summarization models and benchmark factuality metrics, showing their correlation with human judgment as well as their specific strengths and weaknesses.

Key facts. Cited by 285 papers in the map (data through 7 Oct 2026). Builds on 22 of its 43 references that are in the map. First posted on arXiv 27 Apr 2021.

Builds on

22 of 43 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  8. 2015Teaching Machines to Read and Comprehendcited by 1,039
  9. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  11. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  12. 2019Text Summarization with Pretrained Encoderscited by 553
10 more
  1. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 394
  2. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  3. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  4. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  5. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  6. 2018Bottom-Up Abstractive Summarizationcited by 177
  7. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  8. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  9. 2019Assessing The Factual Accuracy of Generated Textcited by 103
  10. 2020StructSum: Summarization via Structured Representationscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 67; 2023: 72; 2024: 66; 2025: 30; 2026: 34.

285 papers.

Sort:

By area:

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  4. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  5. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  6. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  7. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  8. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  9. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  10. 2022Teaching Small Language Models to Reasoncited by 279
  11. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  12. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
38 more
  1. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  2. 2022SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Controlcited by 191
  3. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  4. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  5. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  6. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  7. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  8. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  9. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  10. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 103
  11. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  12. 2023Enhancing Uncertainty-Based Hallucination Detection with Stronger Focuscited by 98
  13. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  14. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  15. 2022Evaluating Human-Language Model Interactioncited by 90
  16. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  17. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  18. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  19. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  20. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75
  21. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  22. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  23. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  24. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  25. 2021Controlled Text Generation as Continuous Optimization with Multiple Constraintscited by 70
  26. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  27. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  28. 2023FELM: Benchmarking Factuality Evaluation of Large Language Modelscited by 66
  29. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  30. 2022Gradient-Based Constrained Sampling from Language Modelscited by 64
  31. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  32. 2023On Early Detection of Hallucinations in Factual Question Answeringcited by 54
  33. 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 50
  34. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  35. 2022FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarizationcited by 49
  36. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  37. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  38. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47

50 most cited of 285 All 285 (JSON)