# BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR

arXiv: https://arxiv.org/abs/2305.13716 · HTML: https://mlsanity.com/paper/2305.13716/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 36 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 28 of 28)

- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [The fifth 'CHiME' Speech Separation and Recognition Challenge: Dataset, task and baselines](https://mlsanity.com/paper/1803.10609/) (2018)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [Continuous speech separation: dataset and analysis](https://mlsanity.com/paper/2001.11482/) (2020)
- [M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challenge](https://mlsanity.com/paper/2110.07393/) (2021)
- [AISHELL-4: An Open Source Dataset for Speech Enhancement, Separation, Recognition and Speaker Diarization in Conference Scenario](https://mlsanity.com/paper/2104.03603/) (2021)
- [MIMO-SPEECH: End-to-End Multi-Channel Multi-Speaker Speech Recognition](https://mlsanity.com/paper/1910.06522/) (2019)
- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [Continuous Speech Separation with Conformer](https://mlsanity.com/paper/2008.05773/) (2020)
- [Recognizing Multi-talker Speech with Permutation Invariant Training](https://mlsanity.com/paper/1704.01985/) (2017)
- [A Purely End-to-end System for Multi-speaker Speech Recognition](https://mlsanity.com/paper/1805.05826/) (2018)
- [End-to-End Monaural Multi-speaker ASR System without Pretraining](https://mlsanity.com/paper/1811.02062/) (2018)
- [CTC Alignments Improve Autoregressive Translation](https://mlsanity.com/paper/2210.05200/) (2022)
- [Single-Channel Multi-talker Speech Recognition with Permutation Invariant Training](https://mlsanity.com/paper/1707.06527/) (2017)
- [Deep Learning based Multi-Source Localization with Source Splitting and its Effectiveness in Multi-Talker Speech Recognition](https://mlsanity.com/paper/2102.07955/) (2021)
- [Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain](https://mlsanity.com/paper/2106.08595/) (2021)
- [A Real-time Speaker Diarization System Based on Spatial Spectrum](https://mlsanity.com/paper/2107.09321/) (2021)
- [Summary On The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Grand Challenge](https://mlsanity.com/paper/2202.03647/) (2022)
- [Simulating realistic speech overlaps improves multi-talker ASR](https://mlsanity.com/paper/2210.15715/) (2022)
- [Sequence to Multi-Sequence Learning via Conditional Chain Mapping for Mixture Signals](https://mlsanity.com/paper/2006.14150/) (2020)
- [An End-to-end Architecture of Online Multi-channel Speech Separation](https://mlsanity.com/paper/2009.03141/) (2020)
- [Graph-PIT: Generalized permutation invariant training for continuous separation of arbitrary numbers of speakers](https://mlsanity.com/paper/2107.14446/) (2021)
- [Speaker-Conditional Chain Model for Speech Separation and Extraction](https://mlsanity.com/paper/2006.14149/) (2020)
- [Location-based training for multi-channel talker-independent speaker separation](https://mlsanity.com/paper/2110.04289/) (2021)
- [MFCCA:Multi-Frame Cross-Channel attention for multi-speaker ASR in Multi-party meeting scenario](https://mlsanity.com/paper/2210.05265/) (2022)

## Built on by (most cited first, 5 of 5)

- [Cross-Speaker Encoding Network for Multi-Talker Speech Recognition](https://mlsanity.com/paper/2401.04152/) (2024)
- [Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTC](https://mlsanity.com/paper/2409.12388/) (2024)
- [Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio](https://mlsanity.com/paper/2505.10975/) (2025)
- [Speaker-Distinguishable CTC: Learning Speaker Distinction Using CTC for Multi-Talker Speech Recognition](https://mlsanity.com/paper/2506.07515/) (2025)
- [End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation](https://mlsanity.com/paper/2311.00697/) (2023)

## Abstract (arXiv)

The recently proposed serialized output training (SOT) simplifies multi-talker automatic speech recognition (ASR) by generating speaker transcriptions separated by a special token. However, frequent speaker changes can make speaker change prediction difficult. To address this, we propose boundary-aware serialized output training (BA-SOT), which explicitly incorporates boundary knowledge into the decoder via a speaker change detection task and boundary constraint loss. We also introduce a two-stage connectionist temporal classification (CTC) strategy that incorporates token-level SOT CTC to restore temporal context information. Besides typical character error rate (CER), we introduce utterance-dependent character error rate (UD-CER) to further measure the precision of speaker change prediction. Compared to original SOT, BA-SOT reduces CER/UD-CER by 5.1%/14.0%, and leveraging a pre-trained ASR model for BA-SOT model initialization further reduces CER/UD-CER by 8.4%/19.9%.