mlsanityData to 7 Oct 2026

FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization

arXiv 2005.03754 ↗First posted 7 May 2020cs.CLcited by 292

Not on the map. Why?

Abstract (arXiv)

Neural abstractive summarization models are prone to generate content inconsistent with the source document, i.e. unfaithful. Existing automatic metrics do not capture such mistakes effectively. We tackle the problem of evaluating faithfulness of a generated summary given its source document. We first collected human annotations of faithfulness for outputs from numerous models on two datasets. We find that current models exhibit a trade-off between abstractiveness and faithfulness: outputs with less word overlap with the source document are more likely to be unfaithful. Next, we propose an automatic question answering (QA) based metric for faithfulness, FEQA, which leverages recent advances in reading comprehension. Given question-answer pairs generated from the summary, a QA model extracts answers from the document; non-matched answers indicate unfaithful information in the summary. Among metrics based on word overlap, embedding similarity, and learned language understanding models, our QA-based metric has significantly higher correlation with human faithfulness scores, especially on highly abstractive summaries.

Key facts. Cited by 292 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 47; 2022: 75; 2023: 55; 2024: 58; 2025: 44; 2026: 13.

292 papers.

Sort:

By area:

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  4. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  5. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  6. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  7. 2023GPTScore: Evaluate as You Desirecited by 357
  8. 2021Efficient Attentions for Long Document Summarizationcited by 342
  9. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  10. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  11. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  12. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
38 more
  1. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  2. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  3. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  4. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  5. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  6. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  7. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  9. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  10. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 200
  11. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  12. 2021Hurdles to Progress in Long-form Question Answeringcited by 166
  13. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  14. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  15. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  16. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  17. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  18. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  19. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  20. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  21. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  22. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  23. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 108
  24. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106
  25. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  26. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  27. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  28. 2022Evaluating Human-Language Model Interactioncited by 90
  29. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  30. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  31. 2021Planning with Learned Entity Prompts for Abstractive Summarizationcited by 75
  32. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  33. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75
  34. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  35. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  36. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  37. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  38. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71

50 most cited of 292 All 292 (JSON)