# CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning

arXiv: https://arxiv.org/abs/2112.08713 · HTML: https://mlsanity.com/paper/2112.08713/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 49 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 30 of 34)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://mlsanity.com/paper/1805.11080/) (2018)
- [Faithful to the Original: Fact Aware Neural Abstractive Summarization](https://mlsanity.com/paper/1711.04434/) (2017)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [SummScreen: A Dataset for Abstractive Screenplay Summarization](https://mlsanity.com/paper/2104.07091/) (2021)
- [Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2010.01672/) (2020)
- [A Survey on Dialogue Summarization: Recent Advances and New Frontiers](https://mlsanity.com/paper/2107.03175/) (2021)
- [Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization](https://mlsanity.com/paper/2105.12544/) (2021)
- [ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining](https://mlsanity.com/paper/2106.00829/) (2021)
- [Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs](https://mlsanity.com/paper/2104.08400/) (2021)
- [Controllable Neural Dialogue Summarization with Personal Named Entity Planning](https://mlsanity.com/paper/2109.13070/) (2021)
- [Coreference-Aware Dialogue Summarization](https://mlsanity.com/paper/2106.08556/) (2021)
- [Controllable Abstractive Dialogue Summarization with Sketch Supervision](https://mlsanity.com/paper/2105.14064/) (2021)
- [EmailSum: Abstractive Email Thread Summarization](https://mlsanity.com/paper/2107.14691/) (2021)
- [Topic-Aware Contrastive Learning for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2109.04994/) (2021)
- [Incorporating Commonsense Knowledge into Abstractive Dialogue Summarization via Heterogeneous Graph Networks](https://mlsanity.com/paper/2010.10044/) (2020)
- [Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation](https://mlsanity.com/paper/2104.07555/) (2021)

## Built on by (most cited first, 30 of 44)

- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity](https://mlsanity.com/paper/2310.07521/) (2023)
- [FaithDial: A Faithful Benchmark for Information-Seeking Dialogue](https://mlsanity.com/paper/2204.10757/) (2022)
- [Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors](https://mlsanity.com/paper/2205.12854/) (2022)
- [TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization](https://mlsanity.com/paper/2402.13249/) (2024)
- [Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites](https://mlsanity.com/paper/2312.01701/) (2023)
- [FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledge](https://mlsanity.com/paper/2305.08281/) (2023)
- [Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articles](https://mlsanity.com/paper/2309.09369/) (2023)
- [Teaching Language Models to Hallucinate Less with Synthetic Tasks](https://mlsanity.com/paper/2310.06827/) (2023)
- [Zero-shot Faithful Factual Error Correction](https://mlsanity.com/paper/2305.07982/) (2023)
- [Whodunit? Learning to Contrast for Authorship Attribution](https://mlsanity.com/paper/2209.11887/) (2022)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [Analyzing and Evaluating Faithfulness in Dialogue Summarization](https://mlsanity.com/paper/2210.11777/) (2022)
- [Improving Fine-tuning of Self-supervised Models with Contrastive Initialization](https://mlsanity.com/paper/2208.00238/) (2022)
- [SWING: Balancing Coverage and Faithfulness for Dialogue Summarization](https://mlsanity.com/paper/2301.10483/) (2023)
- [Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization](https://mlsanity.com/paper/2305.16548/) (2023)
- [Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization](https://mlsanity.com/paper/2402.11414/) (2024)
- [Abstractive Meeting Summarization: A Survey](https://mlsanity.com/paper/2208.04163/) (2022)
- [GersteinLab at MEDIQA-Chat 2023: Clinical Note Summarization from Doctor-Patient Conversations through Fine-tuning and In-context Learning](https://mlsanity.com/paper/2305.05001/) (2023)
- [What are the Desired Characteristics of Calibration Sets? Identifying Correlates on Long Form Scientific Summarization](https://mlsanity.com/paper/2305.07615/) (2023)
- [Key-Element-Informed sLLM Tuning for Document Summarization](https://mlsanity.com/paper/2406.04625/) (2024)
- [Reference Matters: Benchmarking Factual Error Correction for Dialogue Summarization with Fine-grained Evaluation Framework](https://mlsanity.com/paper/2306.05119/) (2023)
- [Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization](https://mlsanity.com/paper/2310.10285/) (2023)
- [MISMATCH: Fine-grained Evaluation of Machine-generated Text with Mismatch Error Types](https://mlsanity.com/paper/2306.10452/) (2023)
- [Generating Faithful Text From a Knowledge Graph with Noisy Reference Text](https://mlsanity.com/paper/2308.06488/) (2023)
- [P^3SUM: Preserving Author's Perspective in News Summarization with Diffusion Language Models](https://mlsanity.com/paper/2311.09741/) (2023)
- [Leveraging Non-dialogue Summaries for Dialogue Summarization](https://mlsanity.com/paper/2210.09474/) (2022)
- [Taxonomy of Abstractive Dialogue Summarization: Scenarios, Approaches and Future Directions](https://mlsanity.com/paper/2210.09894/) (2022)
- [Improving abstractive summarization with energy-based re-ranking](https://mlsanity.com/paper/2210.15553/) (2022)
- [Towards Understanding Omission in Dialogue Summarization](https://mlsanity.com/paper/2211.07145/) (2022)

## Abstract (arXiv)

Factual inconsistencies in generated summaries severely limit the practical applications of abstractive dialogue summarization. Although significant progress has been achieved by using pre-trained models, substantial amounts of hallucinated content are found during the human evaluation. Pre-trained models are most commonly fine-tuned with cross-entropy loss for text summarization, which may not be an optimal strategy. In this work, we provide a typology of factual errors with annotation data to highlight the types of errors and move away from a binary understanding of factuality. We further propose a training strategy that improves the factual consistency and overall quality of summaries via a novel contrastive fine-tuning, called ConFiT. Based on our linguistically-informed typology of errors, we design different modular objectives that each target a specific type. Specifically, we utilize hard negative samples with errors to reduce the generation of factual inconsistency. In order to capture the key information between speakers, we also design a dialogue-specific loss. Using human evaluation and automatic faithfulness metrics, we show that our model significantly reduces all kinds of factual errors on the dialogue summarization, SAMSum corpus. Moreover, our model could be generalized to the meeting summarization, AMI corpus, and it produces significantly higher scores than most of the baselines on both datasets regarding word-overlap metrics.