# Speaker-Distinguishable CTC: Learning Speaker Distinction Using CTC for Multi-Talker Speech Recognition

arXiv: https://arxiv.org/abs/2506.07515 · HTML: https://mlsanity.com/paper/2506.07515/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Builds on (most cited first, 21 of 21)

- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [SpecAugment on Large Scale Datasets](https://mlsanity.com/paper/1912.05533/) (2019)
- [Streaming Multi-Talker ASR with Token-Level Serialized Output Training](https://mlsanity.com/paper/2202.00842/) (2022)
- [End-to-End Speaker-Attributed ASR with Transformer](https://mlsanity.com/paper/2104.02128/) (2021)
- [Recognizing Multi-talker Speech with Permutation Invariant Training](https://mlsanity.com/paper/1704.01985/) (2017)
- [A Purely End-to-end System for Multi-speaker Speech Recognition](https://mlsanity.com/paper/1805.05826/) (2018)
- [Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition System](https://mlsanity.com/paper/2407.09817/) (2024)
- [Cross-Speaker Encoding Network for Multi-Talker Speech Recognition](https://mlsanity.com/paper/2401.04152/) (2024)
- [Data Augmentation For Children's Speech Recognition -- The "Ethiopian" System For The SLT 2021 Children Speech Recognition Challenge](https://mlsanity.com/paper/2011.04547/) (2020)
- [Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition](https://mlsanity.com/paper/2409.00815/) (2024)
- [SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR](https://mlsanity.com/paper/2403.02010/) (2024)
- [A Comparative Study on Speaker-attributed Automatic Speech Recognition in Multi-party Meetings](https://mlsanity.com/paper/2203.16834/) (2022)
- [BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR](https://mlsanity.com/paper/2305.13716/) (2023)
- [Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTC](https://mlsanity.com/paper/2409.12388/) (2024)
- [Alignment-Free Training for Transducer-based Multi-Talker ASR](https://mlsanity.com/paper/2409.20301/) (2024)
- [Speaker Mask Transformer for Multi-talker Overlapped Speech Recognition](https://mlsanity.com/paper/2312.10959/) (2023)

## Built on by (most cited first, 3 of 3)

- [GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASR](https://mlsanity.com/paper/2509.13093/) (2025)
- [Modeling Overlapped Speech with Shuffles](https://mlsanity.com/paper/2603.17769/) (2026)
- [H-SAGE: Holistic Speaker-Aware Guided Experts for MoE-based Multi-Talker ASR](https://mlsanity.com/paper/2607.01566/) (2026)

## Abstract (arXiv)

This paper presents a novel framework for multi-talker automatic speech recognition without the need for auxiliary information. Serialized Output Training (SOT), a widely used approach, suffers from recognition errors due to speaker assignment failures. Although incorporating auxiliary information, such as token-level timestamps, can improve recognition accuracy, extracting such information from natural conversational speech remains challenging. To address this limitation, we propose Speaker-Distinguishable CTC (SD-CTC), an extension of CTC that jointly assigns a token and its corresponding speaker label to each frame. We further integrate SD-CTC into the SOT framework, enabling the SOT model to learn speaker distinction using only overlapping speech and transcriptions. Experimental comparisons show that multi-task learning with SD-CTC and SOT reduces the error rate of the SOT model by 26% and achieves performance comparable to state-of-the-art methods relying on auxiliary information.