# Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognition

arXiv: https://arxiv.org/abs/1906.10876 · HTML: https://mlsanity.com/paper/1906.10876/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jun 2019.

## Built on by (most cited first, 16 of 16)

- [A Review of Speaker Diarization: Recent Advances with Deep Learning](https://mlsanity.com/paper/2101.09624/) (2021)
- [End-to-End Speaker-Attributed ASR with Transformer](https://mlsanity.com/paper/2104.02128/) (2021)
- [DiarizationLM: Speaker Diarization Post-Processing with Large Language Models](https://mlsanity.com/paper/2401.03506/) (2024)
- [Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings](https://mlsanity.com/paper/2203.16685/) (2022)
- [Extending Whisper with prompt tuning to target-speaker ASR](https://mlsanity.com/paper/2312.08079/) (2023)
- [Adapting Multi-Lingual ASR Models for Handling Multiple Talkers](https://mlsanity.com/paper/2305.18747/) (2023)
- [A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio](https://mlsanity.com/paper/2107.02852/) (2021)
- [Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognition](https://mlsanity.com/paper/2409.00815/) (2024)
- [End-to-End Joint Target and Non-Target Speakers ASR](https://mlsanity.com/paper/2306.02273/) (2023)
- [Investigation of Speaker Representation for Target-Speaker Speech Processing](https://mlsanity.com/paper/2410.11243/) (2024)
- [SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper](https://mlsanity.com/paper/2601.19194/) (2026)
- [Cocktail-Talker: Multi-Speaker Dialog Modeling in Noisy Social Environments with Turn Action GRPO](https://mlsanity.com/paper/2607.27756/) (2026)
- [Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Network](https://mlsanity.com/paper/2309.08489/) (2023)
- [Unsupervised Speaker Diarization in Distributed IoT Networks Using Federated Learning](https://mlsanity.com/paper/2404.10842/) (2024)
- [Hypothesis Clustering and Merging: Novel MultiTalker Speech Recognition with Speaker Tokens](https://mlsanity.com/paper/2409.15732/) (2024)
- [Xiaomi-CocktailASR-1 Technical Report](https://mlsanity.com/paper/2609.11274/) (2026)

## Abstract (arXiv)

In this paper, we propose a novel auxiliary loss function for target-speaker automatic speech recognition (ASR). Our method automatically extracts and transcribes target speaker's utterances from a monaural mixture of multiple speakers speech given a short sample of the target speaker. The proposed auxiliary loss function attempts to additionally maximize interference speaker ASR accuracy during training. This will regularize the network to achieve a better representation for speaker separation, thus achieving better accuracy on the target-speaker ASR. We evaluated our proposed method using two-speaker-mixed speech in various signal-to-interference-ratio conditions. We first built a strong target-speaker ASR baseline based on the state-of-the-art lattice-free maximum mutual information. This baseline achieved a word error rate (WER) of 18.06% on the test set while a normal ASR trained with clean data produced a completely corrupted result (WER of 84.71%). Then, our proposed loss further reduced the WER by 6.6% relative to this strong baseline, achieving a WER of 16.87%. In addition to the accuracy improvement, we also showed that the auxiliary output branch for the proposed loss can even be used for a secondary ASR for interference speakers' speech.