# Serialized Output Training by Learned Dominance

arXiv: https://arxiv.org/abs/2407.03966 · HTML: https://mlsanity.com/paper/2407.03966/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 21 of 21)

- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2010.10504/) (2020)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014)
- [Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers](https://mlsanity.com/paper/2006.10930/) (2020)
- [Large-Scale Pre-Training of End-to-End Multi-Talker ASR for Meeting Transcription with Single Distant Microphone](https://mlsanity.com/paper/2103.16776/) (2021)
- [Streaming end-to-end multi-talker speech recognition](https://mlsanity.com/paper/2011.13148/) (2020)
- [Single-Channel Multi-talker Speech Recognition with Permutation Invariant Training](https://mlsanity.com/paper/1707.06527/) (2017)
- [Conformer-based Hybrid ASR System for Switchboard Dataset](https://mlsanity.com/paper/2111.03442/) (2021)
- [A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker One](https://mlsanity.com/paper/2302.09908/) (2023)
- [Cross-Speaker Encoding Network for Multi-Talker Speech Recognition](https://mlsanity.com/paper/2401.04152/) (2024)
- [Investigation of Practical Aspects of Single Channel Speech Separation for ASR](https://mlsanity.com/paper/2107.01922/) (2021)
- [End-to-End Joint Target and Non-Target Speakers ASR](https://mlsanity.com/paper/2306.02273/) (2023)

## Built on by (most cited first, 5 of 5)

- [Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions](https://mlsanity.com/paper/2409.08596/) (2024)
- [Sortformer: A Novel Approach for Permutation-Resolved Speaker Supervision in Speech-to-Text Systems](https://mlsanity.com/paper/2409.06656/) (2024)
- [Disentangling Speakers in Multi-Talker Speech Recognition with Speaker-Aware CTC](https://mlsanity.com/paper/2409.12388/) (2024)
- [Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio](https://mlsanity.com/paper/2505.10975/) (2025)
- [GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASR](https://mlsanity.com/paper/2509.13093/) (2025)

## Abstract (arXiv)

Serialized Output Training (SOT) has showcased state-of-the-art performance in multi-talker speech recognition by sequentially decoding the speech of individual speakers. To address the challenging label-permutation issue, prior methods have relied on either the Permutation Invariant Training (PIT) or the time-based First-In-First-Out (FIFO) rule. This study presents a model-based serialization strategy that incorporates an auxiliary module into the Attention Encoder-Decoder architecture, autonomously identifying the crucial factors to order the output sequence of the speech components in multi-talker speech. Experiments conducted on the LibriSpeech and LibriMix databases reveal that our approach significantly outperforms the PIT and FIFO baselines in both 2-mix and 3-mix scenarios. Further analysis shows that the serialization module identifies dominant speech components in a mixture by factors including loudness and gender, and orders speech components based on the dominance score.