mlsanityData to 7 Oct 2026

QuestEval: Summarization Asks for Fact-based Evaluation

arXiv 2103.12693 ↗First posted 23 Mar 2021cs.CLcited by 241

Not on the map. Why?

Abstract (arXiv)

Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work has proposed evaluation metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source document. Though promising, the proposed approaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QuestEval. In contrast to established metrics such as ROUGE or BERTScore, QuestEval does not require any ground-truth reference. Nonetheless, QuestEval substantially improves the correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance), as shown in the extensive experiments we report.

Key facts. Cited by 241 papers in the map (data through 7 Oct 2026). Builds on 27 of its 42 references that are in the map. First posted on arXiv 23 Mar 2021.

Builds on

27 of 42 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  11. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  12. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
15 more
  1. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  2. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  3. 2018Object Hallucination in Image Captioningcited by 661
  4. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  5. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  6. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 364
  7. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  8. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  9. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  10. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  11. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  12. 2017Neural Question Generation from Text: A Preliminary Studycited by 78
  13. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  14. 2017A Semantic QA-Based Approach for Text Summarization Evaluationcited by 17
  15. 2020Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metricscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 46; 2023: 50; 2024: 63; 2025: 38; 2026: 28.

241 papers.

Sort:

By area:

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  4. 2023GPTScore: Evaluate as You Desirecited by 357
  5. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  6. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  7. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  8. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  9. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  10. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  11. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  12. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
38 more
  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
  3. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  4. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  5. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  6. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  7. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  8. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  9. 2021A Comprehensive Assessment of Dialog Evaluation Metricscited by 90
  10. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  11. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  12. 2022PEER: A Collaborative Language Modelcited by 86
  13. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  14. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  15. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  16. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 80
  17. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  18. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  19. 2023Do Androids Know They're Only Dreaming of Electric Sheep?cited by 67
  20. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  21. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  22. 2023Extractive Summarization via ChatGPT for Faithful Summary Generationcited by 58
  23. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  24. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  25. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  26. 2022SQuALITY: Building a Long-Document Summarization Dataset the Hard Waycited by 50
  27. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  28. 2022FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarizationcited by 49
  29. 2023Detecting and Mitigating Hallucinations in Multilingual Summarisationcited by 48
  30. 2022RHO (ρ): Reducing Hallucination in Open-domain Dialogues with Knowledge Groundingcited by 47
  31. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44
  32. 2022Prompted Opinion Summarization with GPT-3.5cited by 44
  33. 2024Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Modelscited by 44
  34. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  35. 2024A Closer Look at Claim Decompositioncited by 40
  36. 2022InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuningcited by 39
  37. 2021Factual Consistency Evaluation for Text Summarization via Counterfactual Estimationcited by 37
  38. 2022Extractive is not Faithful: An Investigation of Broad Unfaithfulness Problems in Extractive Summarizationcited by 32

50 most cited of 241 All 241 (JSON)