mlsanityData to 14 Oct 2026

Analyzing and Evaluating Faithfulness in Dialogue Summarization

arXiv 2210.11777 ↗First posted 21 Oct 2022cs.CLcited by 12

Not on the map. Why?

Abstract (arXiv)

Dialogue summarization is abstractive in nature, making it suffer from factual errors. The factual correctness of summaries has the highest priority before practical applications. Many efforts have been made to improve faithfulness in text summarization. However, there is a lack of systematic study on dialogue summarization systems. In this work, we first perform the fine-grained human analysis on the faithfulness of dialogue summaries and observe that over 35% of generated summaries are faithfully inconsistent respective the source dialogues. Furthermore, we present a new model-level faithfulness evaluation method. It examines generation models with multi-choice questions created by rule-based transformations. Experimental results show that our evaluation schema is a strong proxy for the factual correctness of summarization models. The human-annotated faithfulness samples and the evaluation toolkit are released to facilitate future research toward faithful dialogue summarization.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 21 Oct 2022.

Builds on

40 of 55 references.

References · 40

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  7. 2015Teaching Machines to Read and Comprehendcited by 1,049
  8. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  10. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  11. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  12. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
28 more
  1. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  2. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  3. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  4. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  5. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  6. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  7. 2020Extractive Summarization as Text Matchingcited by 189
  8. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  9. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  10. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  11. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  12. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  13. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  14. 2020Factual Error Correction for Abstractive Summarization Modelscited by 108
  15. 2021MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarizationcited by 104
  16. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  17. 2020Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarizationcited by 72
  18. 2020What Have We Achieved on Text Summarization?cited by 65
  19. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 56
  20. 2021A Survey on Dialogue Summarization: Recent Advances and New Frontierscited by 51
  21. 2021Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarizationcited by 51
  22. 2021CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuningcited by 44
  23. 2021Coreference-Aware Dialogue Summarizationcited by 34
  24. 2021Topic-Aware Contrastive Learning for Abstractive Dialogue Summarizationcited by 23
  25. 2020Inductive Learning on Commonsense Knowledge Graph Completioncited by 22
  26. 2021TODSum: Task-Oriented Dialogue Summarization with State Trackingcited by 19
  27. 2022CompoundE: Knowledge Graph Embedding with Translation, Rotation and Scaling Compound Operationscited by 10
  28. 2022A Focused Study on Sequence Length for Dialogue Summarizationcited by 6

Built on by

Cited by year

Citing papers per year: 2023: 5; 2024: 4; 2025: 3.

12 papers.

  1. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  2. 2023Towards Unifying Multi-Lingual and Cross-Lingual Summarizationcited by 15
  3. 2024Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluationcited by 11
  4. 2024A review of faithfulness metrics for hallucination assessment in Large Language Modelscited by 9
  5. 2023Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Frameworkcited by 7
  6. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  7. 2023PSentScore: Evaluating Sentiment Polarity in Dialogue Summarizationcited by 4
  8. 2023Instructive Dialogue Summarization with Query Aggregationscited by 2
  9. 2024Increasing faithfulness in human-human dialog summarization with Spoken Language Understanding taskscited by 2
  10. 2025What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarizationcited by 1
  11. 2025Temporal Referential Consistency: Do LLMs Favor Sequences Over Absolute Time References?cited by 1
  12. 2025FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcriptscited by 0