# Speech Recognition and Multi-Speaker Diarization of Long Conversations

arXiv: https://arxiv.org/abs/2005.08072 · HTML: https://mlsanity.com/paper/2005.08072/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2020.

## Built on by (most cited first, 23 of 23)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [End-to-End Speaker-Attributed ASR with Transformer](https://mlsanity.com/paper/2104.02128/) (2021)
- [Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings](https://mlsanity.com/paper/2203.16685/) (2022)
- [Streaming Multi-talker Speech Recognition with Joint Speaker Identification](https://mlsanity.com/paper/2104.02109/) (2021)
- [BERTraffic: BERT-based Joint Speaker Role and Speaker Change Detection for Air Traffic Control Communications](https://mlsanity.com/paper/2110.05781/) (2021)
- [A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio](https://mlsanity.com/paper/2107.02852/) (2021)
- [HANSEN: Human and AI Spoken Text Benchmark for Authorship Analysis](https://mlsanity.com/paper/2310.16746/) (2023)
- [Hypothesis Stitcher for End-to-End Speaker-attributed ASR on Long-form Multi-talker Recordings](https://mlsanity.com/paper/2101.01853/) (2021)
- [Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator](https://mlsanity.com/paper/2305.16263/) (2023)
- [Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder](https://mlsanity.com/paper/2508.20474/) (2025)
- [Unified Autoregressive Modeling for Joint End-to-End Multi-Talker Overlapped Speech Recognition and Speaker Attribute Estimation](https://mlsanity.com/paper/2107.01549/) (2021)
- [Target conversation extraction: Source separation using turn-taking dynamics](https://mlsanity.com/paper/2407.11277/) (2024)
- [Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio](https://mlsanity.com/paper/2505.10975/) (2025)
- [Far from the Shallow: Brain-Predictive Reasoning Embedding through Residual Disentanglement](https://mlsanity.com/paper/2510.22860/) (2025)
- [SDBench: A Comprehensive Benchmark Suite for Speaker Diarization](https://mlsanity.com/paper/2507.16136/) (2025)
- [TOLD: A Novel Two-Stage Overlap-Aware Framework for Speaker Diarization](https://mlsanity.com/paper/2303.05397/) (2023)
- [Proactive Hearing Assistants that Isolate Egocentric Conversations](https://mlsanity.com/paper/2511.11473/) (2025)
- [Unsupervised Speaker Diarization that is Agnostic to Language, Overlap-Aware, and Tuning Free](https://mlsanity.com/paper/2207.12504/) (2022)
- [Speech Diarization and ASR with GMM](https://mlsanity.com/paper/2307.05637/) (2023)
- [Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens](https://mlsanity.com/paper/2607.29363/) (2026)
- [Leading-Silence Augmentation and Multi-Stage Synthetic Supervision for the Second MLC-SLM Challenge](https://mlsanity.com/paper/2608.14150/) (2026)

## Abstract (arXiv)

Speech recognition (ASR) and speaker diarization (SD) models have traditionally been trained separately to produce rich conversation transcripts with speaker labels. Recent advances have shown that joint ASR and SD models can learn to leverage audio-lexical inter-dependencies to improve word diarization performance. We introduce a new benchmark of hour-long podcasts collected from the weekly This American Life radio program to better compare these approaches when applied to extended multi-speaker conversations. We find that training separate ASR and SD models perform better when utterance boundaries are known but otherwise joint models can perform better. To handle long conversations with unknown utterance boundaries, we introduce a striding attention decoding algorithm and data augmentation techniques which, combined with model pre-training, improves ASR and SD.