# Annotating and Detecting Fine-grained Factual Errors for Dialogue Summarization

arXiv: https://arxiv.org/abs/2305.16548 · HTML: https://mlsanity.com/paper/2305.16548/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 55 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 30 of 41)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021)
- [Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization](https://mlsanity.com/paper/2109.09784/) (2021)
- [Evaluating Factuality in Generation with Dependency-level Entailment](https://mlsanity.com/paper/2010.05478/) (2020)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2109.09209/) (2021)
- [Factual Error Correction for Abstractive Summarization Models](https://mlsanity.com/paper/2010.08712/) (2020)
- [Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors](https://mlsanity.com/paper/2205.12854/) (2022)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://mlsanity.com/paper/1909.01610/) (2019)
- [Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection](https://mlsanity.com/paper/2104.09061/) (2021)
- [Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2010.01672/) (2020)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Multi-Fact Correction in Abstractive Text Summarization](https://mlsanity.com/paper/2010.02443/) (2020)

## Built on by (most cited first, 10 of 10)

- [Teaching Large Language Models to Maintain Contextual Faithfulness via Synthetic Tasks and Reinforcement Learning](https://mlsanity.com/paper/2505.16483/) (2025)
- [Face4RAG: Factual Consistency Evaluation for Retrieval Augmented Generation in Chinese](https://mlsanity.com/paper/2407.01080/) (2024)
- [Factual Dialogue Summarization via Learning from Large Language Models](https://mlsanity.com/paper/2406.14709/) (2024)
- [Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy](https://mlsanity.com/paper/2402.12821/) (2024)
- [Learning to Verify Summary Facts with Fine-Grained LLM Feedback](https://mlsanity.com/paper/2412.10689/) (2024)
- [A Critical Look at Meta-evaluating Summarisation Evaluation Metrics](https://mlsanity.com/paper/2409.19507/) (2024)
- [What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization](https://mlsanity.com/paper/2505.12474/) (2025)
- [Zero-shot Factual Consistency Evaluation Across Domains](https://mlsanity.com/paper/2408.04114/) (2024)
- [Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies](https://mlsanity.com/paper/2605.29712/) (2026)
- [Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarization](https://mlsanity.com/paper/2608.09043/) (2026)

## Abstract (arXiv)

A series of datasets and models have been proposed for summaries generated for well-formatted documents such as news articles. Dialogue summaries, however, have been under explored. In this paper, we present the first dataset with fine-grained factual error annotations named DIASUMFACT. We define fine-grained factual error detection as a sentence-level multi-label classification problem, and we evaluate two state-of-the-art (SOTA) models on our dataset. Both models yield sub-optimal results, with a macro-averaged F1 score of around 0.25 over 6 error classes. We further propose an unsupervised model ENDERANKER via candidate ranking using pretrained encoder-decoder models. Our model performs on par with the SOTA models while requiring fewer resources. These observations confirm the challenges in detecting factual errors from dialogue summaries, which call for further studies, for which our dataset and results offer a solid foundation.