# Contextual Adapters for Personalized Speech Recognition in Neural Transducers

arXiv: https://arxiv.org/abs/2205.13660 · HTML: https://mlsanity.com/paper/2205.13660/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 26 references that are in the map.
- First posted on arXiv 26 May 2022.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Efficient parametrization of multi-domain deep neural networks](https://mlsanity.com/paper/1803.10082/) (2018)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusion](https://mlsanity.com/paper/2104.02194/) (2021)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [Context-Aware Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2111.03250/) (2021)
- [Personalization Strategies for End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2102.07739/) (2021)
- [No Need for a Lexicon? Evaluating the Value of the Pronunciation Lexica in End-to-End Models](https://mlsanity.com/paper/1712.01864/) (2017)

## Built on by (most cited first, 30 of 57)

- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [End-to-End Speech Recognition Contextualization with Large Language Models](https://mlsanity.com/paper/2309.10917/) (2023)
- [Robust Acoustic and Semantic Contextual Biasing in Neural Transducers for Speech Recognition](https://mlsanity.com/paper/2305.05271/) (2023)
- [SeACo-Paraformer: A Non-Autoregressive ASR System with Flexible and Effective Hotword Customization Ability](https://mlsanity.com/paper/2308.03266/) (2023)
- [Improving ASR Contextual Biasing with Guided Attention](https://mlsanity.com/paper/2401.08835/) (2024)
- [CopyNE: Better Contextual ASR by Copying Named Entities](https://mlsanity.com/paper/2305.12839/) (2023)
- [Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning](https://mlsanity.com/paper/2307.10274/) (2023)
- [Phoneme-aware Encoding for Prefix-tree-based Contextual ASR](https://mlsanity.com/paper/2312.09582/) (2023)
- [CB-Conformer: Contextual biasing Conformer for biased word recognition](https://mlsanity.com/paper/2304.09607/) (2023)
- [Improving Neural Biasing for Contextual Speech Recognition by Early Context Injection and Text Perturbation](https://mlsanity.com/paper/2407.10303/) (2024)
- [Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Loss](https://mlsanity.com/paper/2406.16120/) (2024)
- [Approximate Nearest Neighbour Phrase Mining for Contextual Speech Recognition](https://mlsanity.com/paper/2304.08862/) (2023)
- [Retrieve and Copy: Scaling ASR Personalization to Large Catalogs](https://mlsanity.com/paper/2311.08402/) (2023)
- [Improving Large-scale Deep Biasing with Phoneme Features and Text-only Data in Streaming Transducer](https://mlsanity.com/paper/2311.08966/) (2023)
- [Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognition](https://mlsanity.com/paper/2301.02736/) (2023)
- [Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimation](https://mlsanity.com/paper/2305.17846/) (2023)
- [A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2309.09552/) (2023)
- [Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASR](https://mlsanity.com/paper/2404.10180/) (2024)
- [Keyword-Guided Adaptation of Automatic Speech Recognition](https://mlsanity.com/paper/2406.02649/) (2024)
- [On-the-fly Text Retrieval for End-to-End ASR Adaptation](https://mlsanity.com/paper/2303.10942/) (2023)
- [Dialog act guided contextual adapter for personalized speech recognition](https://mlsanity.com/paper/2303.17799/) (2023)
- [Multilingual Contextual Adapters To Improve Custom Word Recognition In Low-resource Languages](https://mlsanity.com/paper/2307.00759/) (2023)
- [Exploration of Adapter for Noise Robust Automatic Speech Recognition](https://mlsanity.com/paper/2402.18275/) (2024)
- [DANCER: Entity Description Augmented Named Entity Corrector for Automatic Speech Recognition](https://mlsanity.com/paper/2403.17645/) (2024)
- [Multi-Modal Retrieval For Large Language Model Based Speech Recognition](https://mlsanity.com/paper/2406.09618/) (2024)
- [Leveraging Pretrained ASR Encoders for Effective and Efficient End-to-End Speech Intent Classification and Slot Filling](https://mlsanity.com/paper/2307.07057/) (2023)
- [Promptformer: Prompted Conformer Transducer for ASR](https://mlsanity.com/paper/2401.07360/) (2024)
- [Locality enhanced dynamic biasing and sampling strategies for contextual ASR](https://mlsanity.com/paper/2401.13146/) (2024)
- [Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrieval](https://mlsanity.com/paper/2411.00664/) (2024)
- [WCTC-Biasing: Retraining-free Contextual Biasing ASR with Wildcard CTC-based Keyword Spotting and Inter-layer Biasing](https://mlsanity.com/paper/2506.01263/) (2025)

## Abstract (arXiv)

Personal rare word recognition in end-to-end Automatic Speech Recognition (E2E ASR) models is a challenge due to the lack of training data. A standard way to address this issue is with shallow fusion methods at inference time. However, due to their dependence on external language models and the deterministic approach to weight boosting, their performance is limited. In this paper, we propose training neural contextual adapters for personalization in neural transducer based ASR models. Our approach can not only bias towards user-defined words, but also has the flexibility to work with pretrained ASR models. Using an in-house dataset, we demonstrate that contextual adapters can be applied to any general purpose pretrained ASR model to improve personalization. Our method outperforms shallow fusion, while retaining functionality of the pretrained models by not altering any of the model weights. We further show that the adapter style training is superior to full-fine-tuning of the ASR models on datasets with user-defined content.