mlsanityData to 14 Oct 2026

Asking and Answering Questions to Evaluate the Factual Consistency of Summaries

arXiv 2004.04228 ↗First posted 8 Apr 2020cs.CLcited by 378

Not on the map. Why?

Abstract (arXiv)

Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose an automatic evaluation protocol called QAGS (pronounced "kags") that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text.

Key facts. Cited by 378 papers in the map (data through 14 Oct 2026). First posted on arXiv 8 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 53; 2022: 68; 2023: 81; 2024: 83; 2025: 54; 2026: 39.

378 papers.

Sort:

By area:

  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  8. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  9. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  10. 2023GPTScore: Evaluate as You Desirecited by 361
  11. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  12. 2021Efficient Attentions for Long Document Summarizationcited by 345
38 more
  1. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  2. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  3. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  4. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 287
  5. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  6. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  7. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  8. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  9. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  10. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  11. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  12. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  13. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  14. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  15. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  16. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  17. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 208
  18. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
  19. 2023Compressing Context to Enhance Inference Efficiency of Large Language Modelscited by 189
  20. 2021Hurdles to Progress in Long-form Question Answeringcited by 167
  21. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 156
  22. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  23. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  24. 2021A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generationcited by 138
  25. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  26. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  27. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  28. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  29. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  30. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  31. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 120
  32. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  33. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 109
  34. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 108
  35. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106
  36. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  37. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  38. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96

50 most cited of 378 All 378 (JSON)