mlsanityData to 14 Oct 2026

QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

arXiv 2112.08542 ↗First posted 16 Dec 2021cs.CLcited by 236

Not on the map. Why?

Abstract (arXiv)

Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFactEval, that leads to a 14% average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.

Key facts. Cited by 236 papers in the map (data through 14 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

44 of 60 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  11. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  12. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
32 more
  1. 2015Teaching Machines to Read and Comprehendcited by 1,049
  2. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  3. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  5. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  7. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  8. 2019Text Summarization with Pretrained Encoderscited by 556
  9. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 476
  10. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  11. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  12. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  13. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  14. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  15. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 225
  16. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  17. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  18. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  19. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  20. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  21. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  22. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 82
  23. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 76
  24. 2021DocNLI: A Large-scale Dataset for Document-level Natural Language Inferencecited by 75
  25. 2020What Have We Achieved on Text Summarization?cited by 65
  26. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  27. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  28. 2021MixQG: Neural Question Generation with Mixed Answer Typescited by 28
  29. 2020FFCI: A Framework for Interpretable Automatic Evaluation of Summarizationcited by 25
  30. 2021Question Answering Infused Pre-training of General-Purpose Contextualized Representationscited by 19
  31. 2020SacreROUGE: An Open-Source Library for Using and Developing Summarization Evaluation Metricscited by 18
  32. 2020Adversarial NLI for Factual Correctness in Text Summarisation Modelscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 22; 2023: 53; 2024: 76; 2025: 47; 2026: 37.

236 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  4. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  5. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  6. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  7. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
  8. 2023FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarioscited by 251
  9. 2023Large Language Model Alignment: A Surveycited by 243
  10. 2023Conformal Language Modelingcited by 170
  11. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  12. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 149
38 more
  1. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  2. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  3. 2024Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Mapscited by 142
  4. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  5. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  6. 2023Generating Benchmarks for Factuality Evaluation of Language Modelscited by 105
  7. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  8. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 97
  9. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  10. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  11. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  12. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 67
  13. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
  14. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  15. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  16. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  17. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 49
  18. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  19. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  20. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  21. 2023LitCab: Lightweight Language Model Calibration over Short- and Long-form Responsescited by 44
  22. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 44
  23. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 44
  24. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  25. 2024A Closer Look at Claim Decompositioncited by 41
  26. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  27. 2023Beyond the Chat: Executable and Verifiable Text-Editing with LLMscited by 35
  28. 2024Fewer Truncations Improve Language Modelingcited by 35
  29. 2024Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recallcited by 32
  30. 2024FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extractioncited by 28
  31. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
  32. 2023Zero-shot Faithful Factual Error Correctioncited by 26
  33. 2024WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queriescited by 26
  34. 2024Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trendscited by 25
  35. 2023Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluationcited by 24
  36. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Taskscited by 24
  37. 2023Are Large Language Models Reliable Judges? A Study on the Factuality Evaluation Capabilities of LLMscited by 24
  38. 2024Learning to Summarize from LLM-generated Feedbackcited by 24

50 most cited of 236 All 236 (JSON)