# Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech

arXiv: https://arxiv.org/abs/2605.17652 · HTML: https://mlsanity.com/paper/2605.17652/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 44 references that are in the map.
- First posted on arXiv 17 May 2026.

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [BARTScore: Evaluating Generated Text as Text Generation](https://mlsanity.com/paper/2106.11520/) (2021)
- [Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper)](https://mlsanity.com/paper/1906.01815/) (2019)
- [Do LLMs write like humans? Variation in grammatical and rhetorical styles](https://mlsanity.com/paper/2410.16107/) (2024)
- [Understanding Iterative Revision from Human-Written Text](https://mlsanity.com/paper/2203.03802/) (2022)
- [Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation](https://mlsanity.com/paper/2509.08825/) (2025)
- [Topic-aware Pointer-Generator Networks for Summarizing Spoken Conversations](https://mlsanity.com/paper/1910.01335/) (2019)
- [Generalization Bias in Large Language Model Summarization of Scientific Research](https://mlsanity.com/paper/2504.00025/) (2025)
- [Benchmarking Large Language Model Capabilities for Conditional Generation](https://mlsanity.com/paper/2306.16793/) (2023)
- [Towards Abstractive Grounded Summarization of Podcast Transcripts](https://mlsanity.com/paper/2203.11425/) (2022)
- [Systematic Bias in Large Language Models: Discrepant Response Patterns in Binary vs. Continuous Judgment Tasks](https://mlsanity.com/paper/2504.19445/) (2025)
- [Speech vs. Transcript: Does It Matter for Human Annotators in Speech Summarization?](https://mlsanity.com/paper/2408.07277/) (2024)
- [ESSumm: Extractive Speech Summarization from Untranscribed Meeting](https://mlsanity.com/paper/2209.06913/) (2022)

## Abstract (arXiv)

There are not enough established benchmarks for the task fo speech summarization. Creating new benchmarks demands human annotation, as LLMs could embed systemic errors and bias into datasets. We test ten annotation workflows varying input modality (audio, transcript, or both) and the inclusion of editing (self or peer-editing) to investigate potential quality tradeoffs from using human annotators to summarize audio. We compare human audio-based summaries to human transcript-based summaries to track the impact of the different information modalities on summary quality. We also compare the human outputs against four LLM benchmarks (three text, one audio) to examine whether human-written summaries are less informative than highly fluent automated outputs. We find that audio-based summaries are less informative and more compressed than transcript summaries. However, iterative peer-editing with audio mitigates this difference, enabling audio-based summaries to be as informative as their transcript counterparts and LLM summaries. These findings validate iterative peer-editing among human annotators for the creation of benchmarks informed by both lexical and prosodic information. This enables crucial dataset collection even in setting where transcripts are unavailable.