# Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR

arXiv: https://arxiv.org/abs/2506.22646 · HTML: https://mlsanity.com/paper/2506.22646/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2025.

## Built on by (most cited first, 8 of 8)

- [SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper](https://mlsanity.com/paper/2601.19194/) (2026)
- [GLAD: Global-Local Aware Dynamic Mixture-of-Experts for Multi-Talker ASR](https://mlsanity.com/paper/2509.13093/) (2025)
- [Segment Length Matters: A Study of Segment Lengths on Audio Fingerprinting Performance](https://mlsanity.com/paper/2601.17690/) (2026)
- [G-STAR: End-to-End Global Speaker-Tracking Attributed Recognition](https://mlsanity.com/paper/2603.10468/) (2026)
- [H-SAGE: Holistic Speaker-Aware Guided Experts for MoE-based Multi-Talker ASR](https://mlsanity.com/paper/2607.01566/) (2026)
- [Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correction](https://mlsanity.com/paper/2608.22196/) (2026)
- [Asymmetric Classifier-Free Guidance for Target-Speaker ASR](https://mlsanity.com/paper/2609.30476/) (2026)
- [Understanding Personalization Gains in Whisper](https://mlsanity.com/paper/2610.17019/) (2026)

## Abstract (arXiv)

We propose a self-speaker adaptation method for streaming multi-talker automatic speech recognition (ASR) that eliminates the need for explicit speaker queries. Unlike conventional approaches requiring target speaker embeddings or enrollment audio, our technique dynamically adapts individual ASR instances through speaker-wise speech activity prediction. The key innovation involves injecting speaker-specific kernels generated via speaker supervision activations into selected ASR encoder layers. This enables instantaneous speaker adaptation to target speakers while handling fully overlapped speech even in a streaming scenario. Experiments show state-of-the-art performance in both offline and streaming scenarios, demonstrating that our self-adaptive method effectively addresses severe speech overlap through streamlined speaker-focused recognition. The results validate the proposed self-speaker adaptation approach as a robust solution for multi-talker ASR under severe overlapping speech conditions.