# Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognition

arXiv: https://arxiv.org/abs/2110.02220 · HTML: https://mlsanity.com/paper/2110.02220/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 5 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020)
- [Learning Associative Inference Using Fast Weight Memory](https://mlsanity.com/paper/2011.07831/) (2020)
- [Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entities](https://mlsanity.com/paper/1912.09251/) (2019)
- [Approximating How Single Head Attention Learns](https://mlsanity.com/paper/2103.07601/) (2021)
- [Toward domain-invariant speech recognition via large scale training](https://mlsanity.com/paper/1808.05312/) (2018)
- [Contextual Speech Recognition with Difficult Negative Training Examples](https://mlsanity.com/paper/1810.12170/) (2018)
- [Personalized Speech recognition on mobile devices](https://mlsanity.com/paper/1603.03185/) (2016)
- [Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction](https://mlsanity.com/paper/1909.02855/) (2019)
- [Neural Semantic Encoders](https://mlsanity.com/paper/1607.04315/) (2016)
- [Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling](https://mlsanity.com/paper/2009.01803/) (2020)

## Built on by (most cited first, 25 of 25)

- [Robust Acoustic and Semantic Contextual Biasing in Neural Transducers for Speech Recognition](https://mlsanity.com/paper/2305.05271/) (2023)
- [Improving ASR Contextual Biasing with Guided Attention](https://mlsanity.com/paper/2401.08835/) (2024)
- [Zero-shot Domain-sensitive Speech Recognition with Prompt-conditioning Fine-tuning](https://mlsanity.com/paper/2307.10274/) (2023)
- [CB-Conformer: Contextual biasing Conformer for biased word recognition](https://mlsanity.com/paper/2304.09607/) (2023)
- [Contextual Biasing with the Knuth-Morris-Pratt Matching Algorithm](https://mlsanity.com/paper/2310.00178/) (2023)
- [Approximate Nearest Neighbour Phrase Mining for Contextual Speech Recognition](https://mlsanity.com/paper/2304.08862/) (2023)
- [Retrieve and Copy: Scaling ASR Personalization to Large Catalogs](https://mlsanity.com/paper/2311.08402/) (2023)
- [CUE Vectors: Modular Training of Language Models Conditioned on Diverse Contextual Signals](https://mlsanity.com/paper/2203.08774/) (2022)
- [Using External Off-Policy Speech-To-Text Mappings in Contextual End-To-End Automated Speech Recognition](https://mlsanity.com/paper/2301.02736/) (2023)
- [A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2309.09552/) (2023)
- [Deferred NAM: Low-latency Top-K Context Injection via Deferred Context Encoding for Non-Streaming ASR](https://mlsanity.com/paper/2404.10180/) (2024)
- [Continual Learning for On-Device Speech Recognition using Disentangled Conformers](https://mlsanity.com/paper/2212.01393/) (2022)
- [Locality enhanced dynamic biasing and sampling strategies for contextual ASR](https://mlsanity.com/paper/2401.13146/) (2024)
- [Text Injection for Neural Contextual Biasing](https://mlsanity.com/paper/2406.02921/) (2024)
- [Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrieval](https://mlsanity.com/paper/2411.00664/) (2024)
- [Improving Code-Switching and Named Entity Recognition in ASR with Speech Editing based Data Augmentation](https://mlsanity.com/paper/2306.08588/) (2023)
- [Adaptive Endpointing with Deep Contextual Multi-armed Bandits](https://mlsanity.com/paper/2303.13407/) (2023)
- [Enhancing the Robustness of Contextual ASR to Varying Biasing Information Volumes Through Purified Semantic Correlation Joint Modeling](https://mlsanity.com/paper/2509.05908/) (2025)
- [A Neural Model for Contextual Biasing Score Learning and Filtering](https://mlsanity.com/paper/2510.23849/) (2025)
- [Incorporating Class-based Language Model for Named Entity Recognition in Factorized Neural Transducer](https://mlsanity.com/paper/2309.07648/) (2023)
- [High-precision Voice Search Query Correction via Retrievable Speech-text Embedings](https://mlsanity.com/paper/2401.04235/) (2024)
- [An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems](https://mlsanity.com/paper/2409.10515/) (2024)
- [H-PRM: A Pluggable Hotword Pre-Retrieval Module for Various Speech Recognition Systems](https://mlsanity.com/paper/2508.18295/) (2025)
- [RASST: Retrieval-Augmented Simultaneous Speech Translation](https://mlsanity.com/paper/2601.22777/) (2026)
- [BaLEEN: Biasing with Latent Encoded Entities for Context-Aware ASR](https://mlsanity.com/paper/2609.36913/) (2026)

## Abstract (arXiv)

Fast contextual adaptation has shown to be effective in improving Automatic Speech Recognition (ASR) of rare words and when combined with an on-device personalized training, it can yield an even better recognition result. However, the traditional re-scoring approaches based on an external language model is prone to diverge during the personalized training. In this work, we introduce a model-based end-to-end contextual adaptation approach that is decoder-agnostic and amenable to on-device personalization. Our on-device simulation experiments demonstrate that the proposed approach outperforms the traditional re-scoring technique by 12% relative WER and 15.7% entity mention specific F1-score in a continues personalization scenario.