# Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entities

arXiv: https://arxiv.org/abs/1912.09251 · HTML: https://mlsanity.com/paper/1912.09251/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2019.

## Built on by (most cited first, 30 of 41)

- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects](https://mlsanity.com/paper/2205.04382/) (2022)
- [TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/2211.09325/) (2022)
- [Factorized Neural Transducer for Efficient Language Model Adaptation](https://mlsanity.com/paper/2110.01500/) (2021)
- [Personalization Strategies for End-to-End Speech Recognition Systems](https://mlsanity.com/paper/2102.07739/) (2021)
- [Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognition](https://mlsanity.com/paper/2110.02220/) (2021)
- [DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification](https://mlsanity.com/paper/2305.15957/) (2023)
- [Efficient Federated Meta-Learning over Multi-Access Wireless Networks](https://mlsanity.com/paper/2108.06453/) (2021)
- [Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition models](https://mlsanity.com/paper/2212.01992/) (2022)
- [CB-Conformer: Contextual biasing Conformer for biased word recognition](https://mlsanity.com/paper/2304.09607/) (2023)
- [The History of Speech Recognition to the Year 2030](https://mlsanity.com/paper/2108.00084/) (2021)
- [APLA: Additional Perturbation for Latent Noise with Adversarial Training Enables Consistency](https://mlsanity.com/paper/2308.12605/) (2023)
- [End-to-end contextual asr based on posterior distribution adaptation for hybrid ctc/attention system](https://mlsanity.com/paper/2202.09003/) (2022)
- [A Simple Baseline for Domain Adaptation in End to End ASR Systems Using Synthetic Data](https://mlsanity.com/paper/2206.13240/) (2022)
- [UserLibri: A Dataset for ASR Personalization Using Only Text](https://mlsanity.com/paper/2207.00706/) (2022)
- [On-Device Personalization of Automatic Speech Recognition Models for Disordered Speech](https://mlsanity.com/paper/2106.10259/) (2021)
- [Retraining-free Customized ASR for Enharmonic Words Based on a Named-Entity-Aware Model and Phoneme Similarity Estimation](https://mlsanity.com/paper/2305.17846/) (2023)
- [CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators](https://mlsanity.com/paper/2407.06141/) (2024)
- [FedSPD: A Soft-clustering Approach for Personalized Decentralized Federated Learning](https://mlsanity.com/paper/2410.18862/) (2024)
- [Bristle: Decentralized Federated Learning in Byzantine, Non-i.i.d. Environments](https://mlsanity.com/paper/2110.11006/) (2021)
- [Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunities](https://mlsanity.com/paper/2304.13149/) (2023)
- [SpellMapper: A non-autoregressive neural spellchecker for ASR customization with candidate retrieval based on n-gram mappings](https://mlsanity.com/paper/2306.02317/) (2023)
- [Locality enhanced dynamic biasing and sampling strategies for contextual ASR](https://mlsanity.com/paper/2401.13146/) (2024)
- [BERT WEAVER: Using WEight AVERaging to enable lifelong learning for transformer-based models in biomedical semantic search engines](https://mlsanity.com/paper/2202.10101/) (2022)
- [Improved Factorized Neural Transducer Model For text-only Domain Adaptation](https://mlsanity.com/paper/2309.09524/) (2023)
- [The Gift of Feedback: Improving ASR Model Quality by Learning from User Corrections through Federated Learning](https://mlsanity.com/paper/2310.00141/) (2023)
- [Learning Correspondence for Deformable Objects](https://mlsanity.com/paper/2405.08996/) (2024)
- [Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions](https://mlsanity.com/paper/2405.16184/) (2024)
- [Homogeneous Speaker Features for On-the-Fly Dysarthric and Elderly Speaker Adaptation](https://mlsanity.com/paper/2407.06310/) (2024)
- [CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty](https://mlsanity.com/paper/2412.10431/) (2024)

## Abstract (arXiv)

We study the effectiveness of several techniques to personalize end-to-end speech models and improve the recognition of proper names relevant to the user. These techniques differ in the amounts of user effort required to provide supervision, and are evaluated on how they impact speech recognition performance. We propose using keyword-dependent precision and recall metrics to measure vocabulary acquisition performance. We evaluate the algorithms on a dataset that we designed to contain names of persons that are difficult to recognize. Therefore, the baseline recall rate for proper names in this dataset is very low: 2.4%. A data synthesis approach we developed brings it to 48.6%, with no need for speech input from the user. With speech input, if the user corrects only the names, the name recall rate improves to 64.4%. If the user corrects all the recognition errors, we achieve the best recall of 73.5%. To eliminate the need to upload user data and store personalized models on a server, we focus on performing the entire personalization workflow on a mobile device.