# Towards Personalization of CTC Speech Recognition Models with Contextual Adapters and Adaptive Boosting

arXiv: https://arxiv.org/abs/2210.09510 · HTML: https://mlsanity.com/paper/2210.09510/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation](https://mlsanity.com/paper/2101.00390/) (2021)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Deep context: end-to-end contextual speech recognition](https://mlsanity.com/paper/1808.02480/) (2018)
- [Recent Developments on ESPnet Toolkit Boosted by Conformer](https://mlsanity.com/paper/2010.13956/) (2020)
- [WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2102.01547/) (2021)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Contextual RNN-T For Open Domain ASR](https://mlsanity.com/paper/2006.03411/) (2020)
- [Deep Shallow Fusion for RNN-T Personalization](https://mlsanity.com/paper/2011.07754/) (2020)
- [On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition](https://mlsanity.com/paper/2005.14327/) (2020)
- [Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability](https://mlsanity.com/paper/2007.15188/) (2020)
- [Contextual Adapters for Personalized Speech Recognition in Neural Transducers](https://mlsanity.com/paper/2205.13660/) (2022)
- [Personalization Strategies for End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2102.07739/) (2021)
- [Exploring Transformers for Large-Scale Speech Recognition](https://mlsanity.com/paper/2005.09684/) (2020)
- [Personalized Speech recognition on mobile devices](https://mlsanity.com/paper/1603.03185/) (2016)
- [Listen Attentively, and Spell Once: Whole Sentence Generation via a Non-Autoregressive Architecture for Low-Latency Speech Recognition](https://mlsanity.com/paper/2005.04862/) (2020)
- [On Addressing Practical Challenges for RNN-Transducer](https://mlsanity.com/paper/2105.00858/) (2021)
- [Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition](https://mlsanity.com/paper/1901.10055/) (2019)
- [No Need for a Lexicon? Evaluating the Value of the Pronunciation Lexica in End-to-End Models](https://mlsanity.com/paper/1712.01864/) (2017)
- [A review of on-device fully neural end-to-end automatic speech recognition algorithms](https://mlsanity.com/paper/2012.07974/) (2020)
- [Adapting Long Context NLM for ASR Rescoring in Conversational Agents](https://mlsanity.com/paper/2104.11070/) (2021)
- [G2G: TTS-Driven Pronunciation Learning for Graphemic Hybrid ASR](https://mlsanity.com/paper/1910.12612/) (2019)
- [Space-Efficient Representation of Entity-centric Query Language Models](https://mlsanity.com/paper/2206.14885/) (2022)
- [Letter-Based Speech Recognition with Gated ConvNets](https://mlsanity.com/paper/1712.09444/) (2017)

## Abstract (arXiv)

End-to-end speech recognition models trained using joint Connectionist Temporal Classification (CTC)-Attention loss have gained popularity recently. In these models, a non-autoregressive CTC decoder is often used at inference time due to its speed and simplicity. However, such models are hard to personalize because of their conditional independence assumption that prevents output tokens from previous time steps to influence future predictions. To tackle this, we propose a novel two-way approach that first biases the encoder with attention over a predefined list of rare long-tail and out-of-vocabulary (OOV) words and then uses dynamic boosting and phone alignment network during decoding to further bias the subword predictions. We evaluate our approach on open-source VoxPopuli and in-house medical datasets to showcase a 60% improvement in F1 score on domain-specific rare words over a strong CTC baseline.