# Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications

arXiv: https://arxiv.org/abs/2403.06570 · HTML: https://mlsanity.com/paper/2403.06570/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 42 references that are in the map.
- First posted on arXiv 11 Mar 2024.

## Builds on (most cited first, 26 of 26)

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario](https://mlsanity.com/paper/2005.07272/) (2020)
- [gpuRIR: A Python Library for Room Impulse Response Simulation with GPU Acceleration](https://mlsanity.com/paper/1810.11359/) (2018)
- [End-to-End Multi-speaker Speech Recognition with Transformer](https://mlsanity.com/paper/2002.03921/) (2020)
- [Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers](https://mlsanity.com/paper/2006.10930/) (2020)
- [End-to-End Speaker-Attributed ASR with Transformer](https://mlsanity.com/paper/2104.02128/) (2021)
- [Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone](https://mlsanity.com/paper/2103.16776/) (2021)
- [A Purely End-to-end System for Multi-speaker Speech Recognition](https://mlsanity.com/paper/1805.05826/) (2018)
- [Investigation of End-To-End Speaker-Attributed ASR for Continuous Multi-Talker Recordings](https://mlsanity.com/paper/2008.04546/) (2020)
- [VoxSRC 2020: The Second VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/2012.06867/) (2020)
- [Advances in Online Audio-Visual Meeting Transcription](https://mlsanity.com/paper/1912.04979/) (2019)
- [Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASR](https://mlsanity.com/paper/2110.03151/) (2021)
- [The CHiME-7 DASR Challenge: Distant Meeting Transcription with Multiple Devices in Diverse Scenarios](https://mlsanity.com/paper/2306.13734/) (2023)
- [Microsoft Speaker Diarization System for the VoxCeleb Speaker Recognition Challenge 2020](https://mlsanity.com/paper/2010.11458/) (2020)
- [VoxSRC 2019: The first VoxCeleb Speaker Recognition Challenge](https://mlsanity.com/paper/1912.02522/) (2019)
- [Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings](https://mlsanity.com/paper/2203.16685/) (2022)
- [Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain](https://mlsanity.com/paper/2106.08595/) (2021)
- [Simulating realistic speech overlaps improves multi-talker ASR](https://mlsanity.com/paper/2210.15715/) (2022)
- [Hypothesis Stitcher for End-to-End Speaker-attributed ASR on Long-form Multi-talker Recordings](https://mlsanity.com/paper/2101.01853/) (2021)
- [A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetings](https://mlsanity.com/paper/2203.16834/) (2022)
- [Fast CRDNN: Towards on Site Training of Mobile Construction Machines](https://mlsanity.com/paper/2006.03169/) (2020)
- [End-to-end Multichannel Speaker-Attributed ASR: Speaker Guided Decoder and Input Feature Analysis](https://mlsanity.com/paper/2310.10106/) (2023)

## Built on by (most cited first, 3 of 3)

- [MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models](https://mlsanity.com/paper/2411.18152/) (2024)
- [Speaker Embeddings to Improve Tracking of Intermittent and Moving Speakers](https://mlsanity.com/paper/2506.19875/) (2025)
- [WhoSaidIt: Human-LLM Collaborative Annotation for Text-Based Multilingual Speaker-Attribute Classification](https://mlsanity.com/paper/2605.26070/) (2026)

## Abstract (arXiv)

Past studies on end-to-end meeting transcription have focused on model architecture and have mostly been evaluated on simulated meeting data. We present a novel study aiming to optimize the use of a Speaker-Attributed ASR (SA-ASR) system in real-life scenarios, such as the AMI meeting corpus, for improved speaker assignment of speech segments. First, we propose a pipeline tailored to real-life applications involving Voice Activity Detection (VAD), Speaker Diarization (SD), and SA-ASR. Second, we advocate using VAD output segments to fine-tune the SA-ASR model, considering that it is also applied to VAD segments during test, and show that this results in a relative reduction of Speaker Error Rate (SER) up to 28%. Finally, we explore strategies to enhance the extraction of the speaker embedding templates used as inputs by the SA-ASR system. We show that extracting them from SD output rather than annotated speaker segments results in a relative SER reduction up to 20%.