# End-to-End Multi-Speaker Speech Recognition using Speaker Embeddings and Transfer Learning

arXiv: https://arxiv.org/abs/1908.04737 · HTML: https://mlsanity.com/paper/1908.04737/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2019.

## Built on by (most cited first, 9 of 9)

- [Investigations on Speech Recognition Systems for Low-Resource Dialectal Arabic-English Code-Switching Speech](https://mlsanity.com/paper/2108.12881/) (2021)
- [Multi-user VoiceFilter-Lite via Attentive Speaker Embedding](https://mlsanity.com/paper/2107.01201/) (2021)
- [End-to-End Joint Target and Non-Target Speakers ASR](https://mlsanity.com/paper/2306.02273/) (2023)
- [Alignment-Free Training for Transducer-based Multi-Talker ASR](https://mlsanity.com/paper/2409.20301/) (2024)
- [Survey of End-to-End Multi-Speaker Automatic Speech Recognition for Monaural Audio](https://mlsanity.com/paper/2505.10975/) (2025)
- [Speaker conditioning of acoustic models using affine transformation for multi-speaker speech recognition](https://mlsanity.com/paper/2111.00320/) (2021)
- [Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO](https://mlsanity.com/paper/2607.27756/) (2026)
- [Closing the Gap between Single-User and Multi-User VoiceFilter-Lite](https://mlsanity.com/paper/2202.12169/) (2022)
- [Combining Contrastive and Non-Contrastive Losses for Fine-Tuning Pretrained Models in Speech Analysis](https://mlsanity.com/paper/2211.01964/) (2022)

## Abstract (arXiv)

This paper presents our latest investigation on end-to-end automatic speech recognition (ASR) for overlapped speech. We propose to train an end-to-end system conditioned on speaker embeddings and further improved by transfer learning from clean speech. This proposed framework does not require any parallel non-overlapped speech materials and is independent of the number of speakers. Our experimental results on overlapped speech datasets show that joint conditioning on speaker embeddings and transfer learning significantly improves the ASR performance.