# kNN For Whisper And Its Effect On Bias And Speaker Adaptation

arXiv: https://arxiv.org/abs/2410.18850 · HTML: https://mlsanity.com/paper/2410.18850/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2024.

## Built on by (most cited first, 1 of 1)

- [SoftStep: Learning Sparse Similarity Powers Deep Neighbor-Based Regression](https://mlsanity.com/paper/2506.08139/) (2025)

## Abstract (arXiv)

Speech recognition performance varies by language, domain, and speaker characteristics such as accent, but fine-tuning a model on any of these categories may lead to catastrophic forgetting. Token-level k nearest neighbor search (kNN), first proposed for neural sequence decoders for natural language generation (NLG) and machine translation (MT), is a non-parametric method that instead adapts using inference-time search in an external datastore, without training the underlying model. We show that Whisper, a transformer end-to-end speech model, benefits from kNN. We investigate the differences between the speech and text setups. We discuss implications for speaker adaptation, and analyze improvements by gender, accent, and age.