mlsanityData to 7 Oct 2026

Evaluating the Factual Consistency of Abstractive Text Summarization

arXiv 1910.12840 ↗First posted 28 Oct 2019cs.CLcited by 578

Not on the map. Why?

Abstract (arXiv)

Currently used metrics for assessing summarization algorithms do not account for whether summaries are factually consistent with source documents. We propose a weakly-supervised, model-based approach for verifying factual consistency and identifying conflicts between source documents and a generated summary. Training data is generated by applying a series of rule-based transformations to the sentences of source documents. The factual consistency model is then trained jointly for three tasks: 1) identify whether sentences remain factually consistent after transformation, 2) extract a span in the source documents to support the consistency prediction, 3) extract a span in the summary sentence that is inconsistent if one exists. Transferring this model to summaries generated by several state-of-the art models reveals that this highly scalable approach substantially outperforms previous models, including those trained with strong supervision using standard datasets for natural language inference and fact checking. Additionally, human evaluation shows that the auxiliary span extraction tasks provide useful assistance in the process of verifying factual consistency.

Key facts. Cited by 578 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 77; 2022: 106; 2023: 133; 2024: 97; 2025: 84; 2026: 81.

578 papers.

Sort:

By area:

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  6. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  10. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  11. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  12. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
38 more
  1. 2021Efficient Attentions for Long Document Summarizationcited by 342
  2. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335
  3. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  4. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  5. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  6. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  7. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  10. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 247
  11. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  12. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  13. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  14. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  15. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  16. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  17. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 196
  18. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  19. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  20. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  21. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 137
  22. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  23. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  24. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  25. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  26. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  27. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  28. 2023Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigationcited by 123
  29. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  30. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  31. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  32. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  33. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106
  34. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  35. 2024Are self-explanations from Large Language Models faithful?cited by 101
  36. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  37. 2023Enhancing Uncertainty-Based Hallucination Detection with Stronger Focuscited by 98
  38. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96

50 most cited of 578 All 578 (JSON)