# What's under the hood: Investigating Automatic Metrics on Meeting Summarization

arXiv: https://arxiv.org/abs/2404.11124 · HTML: https://mlsanity.com/paper/2404.11124/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 17 Apr 2024.

## Builds on (most cited first, 10 of 10)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [SummEval: Re-evaluating Summarization Evaluation](https://mlsanity.com/paper/2007.12626/) (2020)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://mlsanity.com/paper/1909.01610/) (2019)
- [Multi-document Summarization via Deep Learning Techniques: A Survey](https://mlsanity.com/paper/2011.04843/) (2020)
- [A Survey on Dialogue Summarization: Recent Advances and New Frontiers](https://mlsanity.com/paper/2107.03175/) (2021)
- [Investigating Efficiently Extending Transformers for Long Input Summarization](https://mlsanity.com/paper/2208.04347/) (2022)
- [Coreference-Aware Dialogue Summarization](https://mlsanity.com/paper/2106.08556/) (2021)
- [An Exploratory Study on Long Dialogue Summarization: What Works and What's Next](https://mlsanity.com/paper/2109.04609/) (2021)
- [BLEU, METEOR, BERTScore: Evaluation of Metrics Performance in Assessing Critical Translation Errors in Sentiment-oriented Text](https://mlsanity.com/paper/2109.14250/) (2021)
- [Advances in Multi-turn Dialogue Comprehension: A Survey](https://mlsanity.com/paper/2103.03125/) (2021)

## Built on by (most cited first, 9 of 9)

- [Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization](https://mlsanity.com/paper/2410.14545/) (2024)
- [What's Wrong? Refining Meeting Summaries with LLM Feedback](https://mlsanity.com/paper/2407.11919/) (2024)
- [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025)
- [Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator](https://mlsanity.com/paper/2411.18444/) (2024)
- [Leveraging Discourse Structure for Extractive Meeting Summarization](https://mlsanity.com/paper/2405.11055/) (2024)
- [You need to MIMIC to get FAME: Solving Meeting Transcript Scarcity with a Multi-Agent Conversations](https://mlsanity.com/paper/2502.13001/) (2025)
- [Segment-level Tree Search for Long Meeting Document Summarization](https://mlsanity.com/paper/2606.08445/) (2026)
- [Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarization](https://mlsanity.com/paper/2608.09043/) (2026)
- [Retrieved-Span Training for Efficient Query-Focused Meeting Summarization on QMSum](https://mlsanity.com/paper/2609.25028/) (2026)

## Abstract (arXiv)

Meeting summarization has become a critical task considering the increase in online interactions. While new techniques are introduced regularly, their evaluation uses metrics not designed to capture meeting-specific errors, undermining effective evaluation. This paper investigates what the frequently used automatic metrics capture and which errors they mask by correlating automatic metric scores with human evaluations across a broad error taxonomy. We commence with a comprehensive literature review on English meeting summarization to define key challenges like speaker dynamics and contextual turn-taking and error types such as missing information and linguistic inaccuracy, concepts previously loosely defined in the field. We examine the relationship between characteristic challenges and errors by using annotated transcripts and summaries from Transformer-based sequence-to-sequence and autoregressive models from the general summary QMSum dataset. Through experimental validation, we find that different model architectures respond variably to challenges in meeting transcripts, resulting in different pronounced links between challenges and errors. Current default-used metrics struggle to capture observable errors, showing weak to mid-correlations, while a third of the correlations show trends of error masking. Only a subset reacts accurately to specific errors, while most correlations show either unresponsiveness or failure to reflect the error's impact on summary quality.