# Asymmetric Classifier-Free Guidance for Target-Speaker ASR

arXiv: https://arxiv.org/abs/2609.30476 · HTML: https://mlsanity.com/paper/2609.30476/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Trusting Your Evidence: Hallucinate Less with Context-aware Decoding](https://mlsanity.com/paper/2305.14739/) (2023)
- [Permutation Invariant Training of Deep Models for Speaker-Independent Multi-talker Speech Separation](https://mlsanity.com/paper/1607.00325/) (2016)
- [WHAM!: Extending Speech Separation to Noisy Environments](https://mlsanity.com/paper/1907.01160/) (2019)
- [LibriMix: An Open-Source Dataset for Generalizable Speech Separation](https://mlsanity.com/paper/2005.11262/) (2020)
- [CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordings](https://mlsanity.com/paper/2004.09249/) (2020)
- [Serialized Output Training for End-to-End Overlapped Speech Recognition](https://mlsanity.com/paper/2003.12687/) (2020)
- [OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning](https://mlsanity.com/paper/2506.00338/) (2025)
- [Towards Sequence Modeling Alignment between Tokenizer and Autoregressive Model](https://mlsanity.com/paper/2506.05289/) (2025)
- [Extending Whisper with prompt tuning to target-speaker ASR](https://mlsanity.com/paper/2312.08079/) (2023)
- [A Sidecar Separator Can Convert a Single-Talker Speech Recognition System to a Multi-Talker One](https://mlsanity.com/paper/2302.09908/) (2023)
- [Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR](https://mlsanity.com/paper/2506.22646/) (2025)
- [Controlling Human Shape and Pose in Text-to-Image Diffusion Models via Domain Adaptation](https://mlsanity.com/paper/2411.04724/) (2024)
- [SQ-Whisper: Speaker-Querying based Whisper Model for Target-Speaker ASR](https://mlsanity.com/paper/2412.05589/) (2024)
- [Enhancing Intelligibility for Generative Target Speech Extraction via Joint Optimization with Target Speaker ASR](https://mlsanity.com/paper/2501.14477/) (2025)
- [Investigation of Speaker Representation for Target-Speaker Speech Processing](https://mlsanity.com/paper/2410.11243/) (2024)
- [SoftCFG: Uncertainty-guided Stable Guidance for Visual Autoregressive Model](https://mlsanity.com/paper/2510.00996/) (2025)
- [CTC-Seeded Token Edit Refinement for Non-Autoregressive Speech Recognition](https://mlsanity.com/paper/2606.28732/) (2026)
- [Xiaomi-CocktailASR-1 Technical Report](https://mlsanity.com/paper/2609.11274/) (2026)

## Abstract (arXiv)

Target-speaker automatic speech recognition (TS-ASR) must identify and transcribe a desired speaker under varying overlap and noise conditions. These changes alter the acoustic evidence for the target speaker in the speech mixture, motivating inference-time calibration of speaker conditioning. We introduce asymmetric classifier-free guidance (CFG) for TS-ASR using Whisper: the speaker-conditioned branch predicts the target transcript, while the speaker-unconditioned branch predicts serialized multi-speaker transcripts. CFG adjusts the contribution of speaker conditioning during decoding through a single guidance scale. We select a global guidance scale on target-domain development data and train a lightweight encoder-based predictor to adjust it for each utterance, keeping the recognition model fixed. Under domain shifts, our full system achieves relative word error rate (WER) reductions of up to 21.8% over the condition-only baseline, and 5.6% over standard conditional decoding of the same CFG-trained model. Oracle analysis shows that substantially larger WER reductions are possible through utterance-level scale selection and identifies how beneficial adjustments vary with domain shifts.