mlsanityData to 7 Oct 2026

Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization

arXiv 2305.16548 ↗First posted 26 May 2023cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

A series of datasets and models have been proposed for summaries generated for well-formatted documents such as news articles. Dialogue summaries, however, have been under explored. In this paper, we present the first dataset with fine-grained factual error annotations named DIASUMFACT. We define fine-grained factual error detection as a sentence-level multi-label classification problem, and we evaluate two state-of-the-art (SOTA) models on our dataset. Both models yield sub-optimal results, with a macro-averaged F1 score of around 0.25 over 6 error classes. We further propose an unsupervised model ENDERANKER via candidate ranking using pretrained encoder-decoder models. Our model performs on par with the SOTA models while requiring fewer resources. These observations confirm the challenges in detecting factual errors from dialogue summaries, which call for further studies, for which our dataset and results offer a solid foundation.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

41 of 55 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  7. 2015Teaching Machines to Read and Comprehendcited by 1,039
  8. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  12. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
29 more
  1. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  2. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  3. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  4. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  5. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  6. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  7. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  8. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  9. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 128
  10. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  11. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  12. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  13. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  14. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  15. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 75
  16. 2020Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarizationcited by 72
  17. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  18. 2020Multi-Fact Correction in Abstractive Text Summarizationcited by 66
  19. 2020What Have We Achieved on Text Summarization?cited by 65
  20. 2020Generating SOAP Notes from Doctor-Patient Conversations Using Modular Summarization Techniquescited by 64
  21. 2020Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structurescited by 49
  22. 2021Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarizationcited by 46
  23. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44
  24. 2022Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infillingcited by 38
  25. 2021Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphscited by 36
  26. 2021Controllable Neural Dialogue Summarization with Personal Named Entity Planningcited by 36
  27. 2021Focus Attention: Promoting Faithfulness and Diversity in Summarizationcited by 30
  28. 2021Topic-Aware Contrastive Learning for Abstractive Dialogue Summarizationcited by 23
  29. 2020Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoderscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 2; 2026: 2.

10 papers.

  1. 2024Face4RAG: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinesecited by 9
  2. 2025Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learningcited by 7
  3. 2024Factual Dialogue Summarization via Learning from Large Language Modelscited by 6
  4. 2024Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomycited by 5
  5. 2024Learning to Verify Summary Facts with Fine-Grained LLM Feedbackcited by 4
  6. 2024A Critical Look at Meta-evaluating Summarisation Evaluation Metricscited by 2
  7. 2025What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarizationcited by 1
  8. 2024Zero-shot Factual Consistency Evaluation Across Domainscited by 0
  9. 2026Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategiescited by 0
  10. 2026Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarizationcited by 0