mlsanityData to 7 Oct 2026

On Faithfulness and Factuality in Abstractive Summarization

arXiv 2005.00661 ↗First posted 2 May 2020cs.CLcited by 895

Not on the map. Why?

Cited by 58 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

It is well known that the standard likelihood training and approximate decoding objectives in neural text generation models lead to less human-like responses for open-ended tasks such as language modeling and story generation. In this paper we have analyzed limitations of these models for abstractive document summarization and found that these models are highly prone to hallucinate content that is unfaithful to the input document. We conducted a large scale human evaluation of several neural abstractive summarization systems to better understand the types of hallucinations they produce. Our human annotators found substantial amounts of hallucinated content in all model generated summaries. However, our analysis does show that pretrained models are better summarizers not only in terms of raw metrics, i.e., ROUGE, but also in generating faithful and factual summaries as evaluated by humans. Furthermore, we show that textual entailment measures better correlate with faithfulness than standard metrics, potentially leading the way to automatic evaluation metrics as well as training and decoding criteria.

Key facts. Cited by 895 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 86; 2022: 142; 2023: 203; 2024: 142; 2025: 141; 2026: 180.

895 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  8. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  9. 2024OpenAI o1 System Cardcited by 1,220
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  12. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
38 more
  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  2. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  3. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  4. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  5. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  6. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  7. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  9. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  10. 2023Active Retrieval Augmented Generationcited by 506
  11. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  12. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  13. 2021A Survey on Automated Fact-Checkingcited by 376
  14. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  15. 2021Efficient Attentions for Long Document Summarizationcited by 342
  16. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  17. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  18. 2023Chain-of-Verification Reduces Hallucination in Large Language Modelscited by 317
  19. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  20. 2023How Language Model Hallucinations Can Snowballcited by 310
  21. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 308
  22. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  23. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  24. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  25. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  26. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  27. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  28. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  29. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  30. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  31. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  32. 2023Fine-tuning Language Models for Factualitycited by 219
  33. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 211
  34. 2025Why Language Models Hallucinatecited by 205
  35. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  36. 2023A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generationcited by 202
  37. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  38. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200

50 most cited of 895 All 895 (JSON)