# Personalization Strategies for End-to-End Speech Recognition Systems arXiv: https://arxiv.org/abs/2102.07739 ยท HTML: https://mlsanity.com/paper/2102.07739/ ## Key facts - Cited by 26 papers in the map (data through 14 Oct 2026). - Builds on 12 of its 23 references that are in the map. - First posted on arXiv 15 Feb 2021. ## Builds on (most cited first, 12 of 12) - [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018) - [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018) - [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017) - [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019) - [A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognition](https://mlsanity.com/paper/1807.10857/) (2018) - [Class LM and word mapping for contextual biasing in End-to-End ASR](https://mlsanity.com/paper/2007.05609/) (2020) - [Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entities](https://mlsanity.com/paper/1912.09251/) (2019) - [Efficient minimum word error rate training of RNN-Transducer for end-to-end speech recognition](https://mlsanity.com/paper/2007.13802/) (2020) - [End-to-end contextual speech recognition using class language models and a token passing decoder](https://mlsanity.com/paper/1812.02142/) (2018) - [Personalized Speech recognition on mobile devices](https://mlsanity.com/paper/1603.03185/) (2016) - [Optimizing expected word error rate via sampling for speech recognition](https://mlsanity.com/paper/1706.02776/) (2017) - [Domain-aware Neural Language Models for Speech Recognition](https://mlsanity.com/paper/2101.03229/) (2021) ## Built on by (most cited first, 26 of 26) - [Contextual Adapters for Personalized Speech Recognition in Neural Transducers](https://mlsanity.com/paper/2205.13660/) (2022) - [Context-Aware Transformer Transducer for Speech Recognition](https://mlsanity.com/paper/2111.03250/) (2021) - [Robust Acoustic and Semantic Contextual Biasing in Neural Transducers for Speech Recognition](https://mlsanity.com/paper/2305.05271/) (2023) - [Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challenges](https://mlsanity.com/paper/2309.07438/) (2023) - [Phoneme-aware Encoding for Prefix-tree-based Contextual ASR](https://mlsanity.com/paper/2312.09582/) (2023) - [CB-Conformer: Contextual biasing Conformer for biased word recognition](https://mlsanity.com/paper/2304.09607/) (2023) - [A Likelihood Ratio based Domain Adaptation Method for E2E Models](https://mlsanity.com/paper/2201.03655/) (2022) - [A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting](https://mlsanity.com/paper/2309.09552/) (2023) - [On-the-fly Text Retrieval for End-to-End ASR Adaptation](https://mlsanity.com/paper/2303.10942/) (2023) - [Dialog act guided contextual adapter for personalized speech recognition](https://mlsanity.com/paper/2303.17799/) (2023) - [Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunities](https://mlsanity.com/paper/2304.13149/) (2023) - [SpellMapper: A non-autoregressive neural spellchecker for ASR customization with candidate retrieval based on n-gram mappings](https://mlsanity.com/paper/2306.02317/) (2023) - [Multilingual Contextual Adapters To Improve Custom Word Recognition In Low-resource Languages](https://mlsanity.com/paper/2307.00759/) (2023) - [Multi-Modal Retrieval For Large Language Model Based Speech Recognition](https://mlsanity.com/paper/2406.09618/) (2024) - [Can Visual Context Improve Automatic Speech Recognition for an Embodied Agent?](https://mlsanity.com/paper/2210.13189/) (2022) - [A Study on Bias and Fairness In Deep Speaker Recognition](https://mlsanity.com/paper/2303.08026/) (2023) - [Dual-Attention Neural Transducers for Efficient Wake Word Spotting in Speech Recognition](https://mlsanity.com/paper/2304.01905/) (2023) - [Personalization for BERT-based Discriminative Speech Recognition Rescoring](https://mlsanity.com/paper/2307.06832/) (2023) - [Towards Continual Entity Learning in Language Models for Conversational Agents](https://mlsanity.com/paper/2108.00082/) (2021) - [An Effective Context-Balanced Adaptation Approach for Long-Tailed Speech Recognition](https://mlsanity.com/paper/2409.06468/) (2024) - [When Helpful Context Leaks: Privacy Risks in Domain-Adapted ASR](https://mlsanity.com/paper/2605.28211/) (2026) - [Towards Personalization of CTC Speech Recognition Models with Contextual Adapters and Adaptive Boosting](https://mlsanity.com/paper/2210.09510/) (2022) - [Record Deduplication for Entity Distribution Modeling in ASR Transcripts](https://mlsanity.com/paper/2306.06246/) (2023) - [Wiki-En-ASR-Adapt: Large-scale synthetic dataset for English ASR Customization](https://mlsanity.com/paper/2309.17267/) (2023) - [An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems](https://mlsanity.com/paper/2409.10515/) (2024) - [H-PRM: A Pluggable Hotword Pre-Retrieval Module for Various Speech Recognition Systems](https://mlsanity.com/paper/2508.18295/) (2025) ## Abstract (arXiv) The recognition of personalized content, such as contact names, remains a challenging problem for end-to-end speech recognition systems. In this work, we demonstrate how first and second-pass rescoring strategies can be leveraged together to improve the recognition of such words. Following previous work, we use a shallow fusion approach to bias towards recognition of personalized content in the first-pass decoding. We show that such an approach can improve personalized content recognition by up to 16% with minimum degradation on the general use case. We describe a fast and scalable algorithm that enables our biasing models to remain at the word-level, while applying the biasing at the subword level. This has the advantage of not requiring the biasing models to be dependent on any subword symbol table. We also describe a novel second-pass de-biasing approach: used in conjunction with a first-pass shallow fusion that optimizes on oracle WER, we can achieve an additional 14% improvement on personalized content recognition, and even improve accuracy for the general use case by up to 2.5%.