# What Are They Talking About? A Benchmark of Knowledge-Grounded Discussion Summarization

arXiv: https://arxiv.org/abs/2505.12474 · HTML: https://mlsanity.com/paper/2505.12474/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 18 May 2025.

## Builds on (most cited first, 30 of 33)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization](https://mlsanity.com/paper/1911.12237/) (2019)
- [Benchmarking Large Language Models for News Summarization](https://mlsanity.com/paper/2301.13848/) (2023)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation](https://mlsanity.com/paper/2212.07981/) (2022)
- [Structured Chain-of-Thought Prompting for Code Generation](https://mlsanity.com/paper/2305.06599/) (2023)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study](https://mlsanity.com/paper/2304.00723/) (2023)
- [MeetingBank: A Benchmark Dataset for Meeting Summarization](https://mlsanity.com/paper/2305.17529/) (2023)
- [FineSurE: Fine-grained Summarization Evaluation using LLMs](https://mlsanity.com/paper/2407.00908/) (2024)
- [DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarization](https://mlsanity.com/paper/2109.02492/) (2021)
- [Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization](https://mlsanity.com/paper/2311.09184/) (2023)
- [Grounded Conversation Generation as Guided Traverses in Commonsense Knowledge Graphs](https://mlsanity.com/paper/1911.02707/) (2019)
- [Structure-Aware Abstractive Conversation Summarization via Discourse and Action Graphs](https://mlsanity.com/paper/2104.08400/) (2021)
- [FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction](https://mlsanity.com/paper/2403.02270/) (2024)
- [CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization](https://mlsanity.com/paper/2108.13139/) (2021)
- [Analyzing LLM Behavior in Dialogue Summarization: Unveiling Circumstantial Hallucination Trends](https://mlsanity.com/paper/2406.03487/) (2024)
- [Learning to Summarize from LLM-generated Feedback](https://mlsanity.com/paper/2410.13116/) (2024)
- [UniSumEval: Towards Unified, Fine-Grained, Multi-Dimensional Summarization Evaluation for LLMs](https://mlsanity.com/paper/2409.19898/) (2024)
- [Low-Resource Dialogue Summarization with Domain-Agnostic Multi-Source Pretraining](https://mlsanity.com/paper/2109.04080/) (2021)
- [FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document](https://mlsanity.com/paper/2404.11184/) (2024)
- [Other Roles Matter! Enhancing Role-Oriented Dialogue Summarization via Role Interactions](https://mlsanity.com/paper/2205.13190/) (2022)
- [Analyzing and Evaluating Faithfulness in Dialogue Summarization](https://mlsanity.com/paper/2210.11777/) (2022)

## Built on by (most cited first, 1 of 1)

- [A Large-Scale Multi-Dimensional Empirical Study of LLMs for Conversation Summarization](https://mlsanity.com/paper/2606.15974/) (2026)

## Abstract (arXiv)

Traditional dialogue summarization primarily focuses on dialogue content, assuming it comprises adequate information for a clear summary. However, this assumption often fails for discussions grounded in shared background, where participants frequently omit context and use implicit references. This results in summaries that are confusing to readers unfamiliar with the background. To address this, we introduce Knowledge-Grounded Discussion Summarization (KGDS), a novel task that produces a supplementary background summary for context and a clear opinion summary with clarified references. To facilitate research, we construct the first KGDS benchmark, featuring news-discussion pairs and expert-created multi-granularity gold annotations for evaluating sub-summaries. We also propose a novel hierarchical evaluation framework with fine-grained and interpretable metrics. Our extensive evaluation of 12 advanced large language models (LLMs) reveals that KGDS remains a significant challenge. The models frequently miss key facts and retain irrelevant ones in background summarization, and often fail to resolve implicit references in opinion summary integration.