# Target Speaker ASR with Whisper

arXiv: https://arxiv.org/abs/2409.09543 · HTML: https://mlsanity.com/paper/2409.09543/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Sep 2024.

## Built on by (most cited first, 9 of 9)

- [FlowTSE: Target Speaker Extraction with Flow Matching](https://mlsanity.com/paper/2505.14465/) (2025)
- [Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challenges](https://mlsanity.com/paper/2507.18161/) (2025)
- [Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition](https://mlsanity.com/paper/2510.03723/) (2025)
- [SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper](https://mlsanity.com/paper/2601.19194/) (2026)
- [Towards few-shot isolated word reading assessment](https://mlsanity.com/paper/2507.12217/) (2025)
- [SC-SOT: Conditioning the Decoder on Diarized Speaker Information for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2506.12672/) (2025)
- [Whisper-UT: A Unified Translation Framework for Speech and Text](https://mlsanity.com/paper/2509.16375/) (2025)
- [CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASR](https://mlsanity.com/paper/2601.22792/) (2026)
- [Mask2Flow-TSE: Two-Stage Target Speaker Extraction with Masking and Flow Matching](https://mlsanity.com/paper/2603.12837/) (2026)

## Abstract (arXiv)

We propose a novel approach to enable the use of large, single-speaker ASR models, such as Whisper, for target speaker ASR. The key claim of this method is that it is much easier to model relative differences among speakers by learning to condition on frame-level diarization outputs than to learn the space of all speaker embeddings. We find that adding even a single bias term per diarization output type before the first transformer block can transform single-speaker ASR models into target-speaker ASR models. Our approach also supports speaker-attributed ASR by sequentially generating transcripts for each speaker in a diarization output. This simplified method outperforms baseline speech separation and diarization cascade by 12.9 % absolute ORC-WER on the NOTSOFAR-1 dataset.