# Residual Adapters for Parameter-Efficient ASR Adaptation to Atypical and Accented Speech

arXiv: https://arxiv.org/abs/2109.06952 · HTML: https://mlsanity.com/paper/2109.06952/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 30 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Simple, Scalable Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1909.08478/) (2019)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model](https://mlsanity.com/paper/1909.05330/) (2019)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [Personalizing ASR for Dysarthric and Accented Speech with Limited Data](https://mlsanity.com/paper/1907.13511/) (2019)
- [Advancing RNN Transducer Technology for Speech Recognition](https://mlsanity.com/paper/2103.09935/) (2021)
- [Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation](https://mlsanity.com/paper/1601.02828/) (2016)
- [Fast Domain Adaptation for Neural Machine Translation](https://mlsanity.com/paper/1612.06897/) (2016)
- [Parrotron: An End-to-End Speech-to-Speech Conversion Model and its Applications to Hearing-Impaired Speech and Speech Separation](https://mlsanity.com/paper/1904.04169/) (2019)
- [Multi-Accent Adaptation based on Gate Mechanism](https://mlsanity.com/paper/2011.02774/) (2020)
- [Leveraging native language information for improved accented speech recognition](https://mlsanity.com/paper/1904.09038/) (2019)

## Built on by (most cited first, 30 of 31)

- [Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages](https://mlsanity.com/paper/2303.01037/) (2023)
- [A Survey of Resource-efficient LLM and Multimodal Foundation Models](https://mlsanity.com/paper/2401.08092/) (2024)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition](https://mlsanity.com/paper/2301.07851/) (2023)
- [Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts](https://mlsanity.com/paper/2311.01070/) (2023)
- [LatentEditor: Text Driven Local Editing of 3D Scenes](https://mlsanity.com/paper/2312.09313/) (2023)
- [Disentangling Voice and Content with Self-Supervision for Speaker Recognition](https://mlsanity.com/paper/2310.01128/) (2023)
- [RST-LoRA: A Discourse-Aware Low-Rank Adaptation for Long Document Abstractive Summarization](https://mlsanity.com/paper/2405.00657/) (2024)
- [Residual Adapters for Few-Shot Text-to-Speech Speaker Adaptation](https://mlsanity.com/paper/2210.15868/) (2022)
- [Efficient Computation Sharing for Multi-Task Visual Scene Understanding](https://mlsanity.com/paper/2303.09663/) (2023)
- [A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model](https://mlsanity.com/paper/2305.11244/) (2023)
- [Federated Learning for Inference at Anytime and Anywhere](https://mlsanity.com/paper/2212.04084/) (2022)
- [Parameter-Efficient Learning for Text-to-Speech Accent Adaptation](https://mlsanity.com/paper/2305.11320/) (2023)
- [Latent Phrase Matching for Dysarthric Speech](https://mlsanity.com/paper/2306.05446/) (2023)
- [Don't Stop Self-Supervision: Accent Adaptation of Speech Representations via Residual Adapters](https://mlsanity.com/paper/2307.00453/) (2023)
- [Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models](https://mlsanity.com/paper/2403.19709/) (2024)
- [Efficient Compression of Multitask Multilingual Speech Models](https://mlsanity.com/paper/2405.00966/) (2024)
- [Hypernetworks for Personalizing ASR to Atypical Speech](https://mlsanity.com/paper/2406.04240/) (2024)
- [Damage Control During Domain Adaptation for Transducer Based Automatic Speech Recognition](https://mlsanity.com/paper/2210.03255/) (2022)
- [Differentially Private Adapters for Parameter Efficient Acoustic Modeling](https://mlsanity.com/paper/2305.11360/) (2023)
- [Improving Self-supervised Pre-training using Accent-Specific Codebooks](https://mlsanity.com/paper/2407.03734/) (2024)
- [UME: Upcycling Mixture-of-Experts for Scalable and Efficient Automatic Speech Recognition](https://mlsanity.com/paper/2412.17507/) (2024)
- [MoFE: Mixture of Frozen Experts Architecture](https://mlsanity.com/paper/2503.06491/) (2025)
- [Idiosyncratic Versus Normative Modeling of Atypical Speech Recognition: Dysarthric Case Studies](https://mlsanity.com/paper/2509.16718/) (2025)
- [Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognition](https://mlsanity.com/paper/2602.01967/) (2026)
- [A meta learning scheme for fast accent domain expansion in Mandarin speech recognition](https://mlsanity.com/paper/2307.12262/) (2023)
- [Cascaded Multi-task Adaptive Learning Based on Neural Architecture Search](https://mlsanity.com/paper/2310.17664/) (2023)
- [Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR](https://mlsanity.com/paper/2410.13445/) (2024)
- [ACES: Accent Subspaces for Coupling, Explanations, and Stress-Testing in Automatic Speech Recognition](https://mlsanity.com/paper/2603.03359/) (2026)
- [Huntington Disease Automatic Speech Recognition with Biomarker Supervision](https://mlsanity.com/paper/2603.11168/) (2026)

## Abstract (arXiv)

Automatic Speech Recognition (ASR) systems are often optimized to work best for speakers with canonical speech patterns. Unfortunately, these systems perform poorly when tested on atypical speech and heavily accented speech. It has previously been shown that personalization through model fine-tuning substantially improves performance. However, maintaining such large models per speaker is costly and difficult to scale. We show that by adding a relatively small number of extra parameters to the encoder layers via so-called residual adapter, we can achieve similar adaptation gains compared to model fine-tuning, while only updating a tiny fraction (less than 0.5%) of the model parameters. We demonstrate this on two speech adaptation tasks (atypical and accented speech) and for two state-of-the-art ASR architectures.