# Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?

arXiv: https://arxiv.org/abs/2408.07277 · HTML: https://mlsanity.com/paper/2408.07277/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Builds on (most cited first, 23 of 23)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [How2: A Large-scale Dataset for Multimodal Language Understanding](https://mlsanity.com/paper/1811.00347/) (2018)
- [MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://mlsanity.com/paper/2103.06410/) (2021)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [Answers Unite! Unsupervised Metrics for Reinforced Summarization Models](https://mlsanity.com/paper/1909.01610/) (2019)
- [Fill in the BLANC: Human-free quality estimation of document summaries](https://mlsanity.com/paper/2002.09836/) (2020)
- [ChatGPT as a Factual Inconsistency Evaluator for Text Summarization](https://mlsanity.com/paper/2303.15621/) (2023)
- [SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarization](https://mlsanity.com/paper/2005.03724/) (2020)
- [Better Summarization Evaluation with Word Embeddings for ROUGE](https://mlsanity.com/paper/1508.06034/) (2015)
- [Large Language Models are Diverse Role-Players for Summarization Evaluation](https://mlsanity.com/paper/2303.15078/) (2023)
- [Multimodal Abstractive Summarization for How2 Videos](https://mlsanity.com/paper/1906.07901/) (2019)
- [SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Tasks](https://mlsanity.com/paper/2212.10525/) (2022)
- [Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization](https://mlsanity.com/paper/2109.02401/) (2021)
- [Speech Summarization using Restricted Self-Attention](https://mlsanity.com/paper/2110.06263/) (2021)
- [Leveraging Large Text Corpora for End-to-End Speech Summarization](https://mlsanity.com/paper/2303.00978/) (2023)
- [Attention-based Multi-hypothesis Fusion for Speech Summarization](https://mlsanity.com/paper/2111.08201/) (2021)
- [Spoken Language Intent Detection using Confusion2Vec](https://mlsanity.com/paper/1904.03576/) (2019)
- [Transfer Learning from Pre-trained Language Models Improves End-to-End Speech Summarization](https://mlsanity.com/paper/2306.04233/) (2023)
- [BASS: Block-wise Adaptation for Speech Summarization](https://mlsanity.com/paper/2307.08217/) (2023)

## Built on by (most cited first, 3 of 3)

- [Summarizing Speech: A Comprehensive Survey](https://mlsanity.com/paper/2504.08024/) (2025)
- [Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech](https://mlsanity.com/paper/2605.17652/) (2026)
- [VoxSumm: A Multilingual Corpus of Long-Form Spoken News for Joint Summarization and Translation](https://mlsanity.com/paper/2608.10359/) (2026)

## Abstract (arXiv)

Reference summaries for abstractive speech summarization require human annotation, which can be performed by listening to an audio recording or by reading textual transcripts of the recording. In this paper, we examine whether summaries based on annotators listening to the recordings differ from those based on annotators reading transcripts. Using existing intrinsic evaluation based on human evaluation, automatic metrics, LLM-based evaluation, and a retrieval-based reference-free method. We find that summaries are indeed different based on the source modality, and that speech-based summaries are more factually consistent and information-selective than transcript-based summaries. Meanwhile, transcript-based summaries are impacted by recognition errors in the source, and expert-written summaries are more informative and reliable. We make all the collected data and analysis code public(https://github.com/cmu-mlsp/interview_humanssum) to facilitate the reproduction of our work and advance research in this area.