mlsanityData to 7 Oct 2026

Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Framework

arXiv 2306.05119 ↗First posted 8 Jun 2023cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Factuality is important to dialogue summarization. Factual error correction (FEC) of model-generated summaries is one way to improve factuality. Current FEC evaluation that relies on factuality metrics is not reliable and detailed enough. To address this problem, we are the first to manually annotate a FEC dataset for dialogue summarization containing 4000 items and propose FERRANTI, a fine-grained evaluation framework based on reference correction that automatically evaluates the performance of FEC models on different error categories. Using this evaluation framework, we conduct sufficient experiments with FEC approaches under a variety of settings and find the best training modes and significant differences in the performance of the existing approaches on different factual error categories.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 39 of its 65 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

39 of 65 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  6. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  11. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  12. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
27 more
  1. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  2. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  3. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  4. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  5. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 181
  6. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  7. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  8. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  9. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  10. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  11. 2020Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarizationcited by 72
  12. 2020Multi-Fact Correction in Abstractive Text Summarizationcited by 66
  13. 2022FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarizationcited by 49
  14. 2020Evidence-based Factual Error Correctioncited by 44
  15. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44
  16. 2022Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infillingcited by 38
  17. 2021ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Miningcited by 37
  18. 2021Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphscited by 36
  19. 2021Controllable Neural Dialogue Summarization with Personal Named Entity Planningcited by 36
  20. 2021Coreference-Aware Dialogue Summarizationcited by 34
  21. 2022Evaluating Factuality in Text Simplificationcited by 34
  22. 2019Automatic Fact-guided Sentence Modificationcited by 28
  23. 2021Controllable Abstractive Dialogue Summarization with Sketch Supervisioncited by 27
  24. 2020Can Automatic Post-Editing Improve NMT?cited by 17
  25. 2022Improving Factual Consistency in Summarization with Compression-Based Post-Editingcited by 13
  26. 2022Analyzing and Evaluating Faithfulness in Dialogue Summarizationcited by 12
  27. 2022Converge to the Truth: Factual Error Correction via Iterative Constrained Editingcited by 9

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 5.

7 papers.

  1. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  2. 2024Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trendscited by 25
  3. 2024Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Processcited by 11
  4. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 9
  5. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  6. 2024Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomycited by 5
  7. 2023Instructive Dialogue Summarization with Query Aggregationscited by 2