# Understanding Personalization Gains in Whisper

arXiv: https://arxiv.org/abs/2610.17019 · HTML: https://mlsanity.com/paper/2610.17019/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 12 Oct 2026.

## Builds on (most cited first, 16 of 16)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation](https://mlsanity.com/paper/1601.02828/) (2016)
- [LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASR](https://mlsanity.com/paper/2406.06619/) (2024)
- [Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech](https://mlsanity.com/paper/2109.06952/) (2021)
- [Speaker Adaptation for Attention-Based End-to-End Speech Recognition](https://mlsanity.com/paper/1911.03762/) (2019)
- [Text is All You Need: Personalizing ASR Models using Controllable Speech Synthesis](https://mlsanity.com/paper/2303.14885/) (2023)
- [Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR](https://mlsanity.com/paper/2506.22646/) (2025)
- [SAML: Speaker Adaptive Mixture of LoRA Experts for End-to-End ASR](https://mlsanity.com/paper/2406.19706/) (2024)
- [Variational Low-Rank Adaptation for Personalized Impaired Speech Recognition](https://mlsanity.com/paper/2509.20397/) (2025)
- [Hypernetworks for Personalizing ASR to Atypical Speech](https://mlsanity.com/paper/2406.04240/) (2024)
- [PI-Whisper: Designing an Adaptive and Incremental Automatic Speech Recognition System for Edge Devices](https://mlsanity.com/paper/2406.15668/) (2024)
- [Speaker Adaptation for Quantised End-to-End ASR Models](https://mlsanity.com/paper/2408.03979/) (2024)
- [Two-Stage Adaptation for Non-Normative Speech Recognition: Revisiting Speaker-Independent Initialization for Personalization](https://mlsanity.com/paper/2603.15261/) (2026)

## Abstract (arXiv)

Whisper-style ASR models provide strong zero-shot transcription, but their residual errors are not evenly distributed across datasets, recording conditions, and metadata-based groups. We study when lightweight low-rank adaptation (LoRA) improves Whisper and whether observed gains require group-specific adapters. Across Common Voice, LibriSpeech, and VoxPopuli, we compare zero-shot Whisper, pooled LoRA, and group-specific LoRA under matched within-group splits. We find that adaptation is not uniformly beneficial: gains are concentrated in difficult Common Voice groups, are consistent on VoxPopuli, and largely disappear on low-error LibriSpeech. Crucially, pooled LoRA often matches, and sometimes clearly outperforms, group-specific LoRA. Gains over zero-shot should therefore be tested against matched pooled adaptation before being interpreted as evidence of personalization.