mlsanityData to 7 Oct 2026

An Investigation Into On-device Personalization of End-to-end Automatic Speech Recognition Models

arXiv 1909.06678 ↗First posted 14 Sep 2019eess.AS cs.LG cs.SDcited by 25

Not on the map. Why?

Abstract (arXiv)

Speaker-independent speech recognition systems trained with data from many users are generally robust against speaker variability and work well for a large population of speakers. However, these systems do not always generalize well for users with very different speech characteristics. This issue can be addressed by building personalized systems that are designed to work well for each specific user. In this paper, we investigate the idea of securely training personalized end-to-end speech recognition models on mobile devices so that user data and models never leave the device and are never stored on a server. We study how the mobile training environment impacts performance by simulating on-device data consumption. We conduct experiments using data collected from speech impaired users for personalization. Our results show that personalization achieved 63.7% relative word error rate reduction when trained in a server environment and 58.1% in a mobile environment. Moving to on-device personalization resulted in 18.7% performance degradation, in exchange for improved scalability and data privacy. To train the model on device, we split the gradient computation into two and achieved 45% memory reduction at the expense of 42% increase in training time.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 2; 2023: 3; 2024: 1; 2025: 3; 2026: 2.

25 papers.

Sort:

  1. 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 189
  2. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  3. 2021Personalized Federated Learning through Local Memorizationcited by 57
  4. 2021FedSoft: Soft Clustered Federated Learning with Proximal Local Updatingcited by 46
  5. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  6. 2023Personalized Federated Learning under Mixture of Distributionscited by 37
  7. 2021Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneitycited by 27
  8. 2021PFA: Privacy-preserving Federated Adaptation for Effective Model Personalizationcited by 18
  9. 2021Revealing and Protecting Labels in Distributed Trainingcited by 17
  10. 2021The History of Speech Recognition to the Year 2030cited by 10
  11. 2021Teaching keyword spotters to spot new keywords with limited examplescited by 9
  12. 2022Detecting Unintended Memorization in Language-Model-Fused ASRcited by 9
13 more
  1. 2022Personalized Federated Learning via Heterogeneous Modular Networkscited by 8
  2. 2021On-Device Personalization of Automatic Speech Recognition Models for Disordered Speechcited by 5
  3. 2021Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognitioncited by 5
  4. 2023Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunitiescited by 4
  5. 2021Error-driven Fixed-Budget ASR Personalization for Accented Speakerscited by 2
  6. 2021Management of Resource at the Network Edge for Federated Learningcited by 2
  7. 2025Hierarchical Knowledge Structuring for Effective Federated Learning in Heterogeneous Environmentscited by 1
  8. 2023MobileASR: A resource-aware on-device learning framework for user voice personalization applications on mobile phonescited by 0
  9. 2024Personalized Federated Learning for Statistical Heterogeneitycited by 0
  10. 2025Moss: Proxy Model-based Full-Weight Aggregation in Federated Learning with Heterogeneous Modelscited by 0
  11. 2025Adaptive Dual-Mode Distillation with Incentive Schemes for Scalable, Heterogeneous Federated Learning on Non-IID Datacited by 0
  12. 2026Multimodal Consistency-Guided Reference-Free Data Selection for ASR Accent Adaptationcited by 0
  13. 2026Similarity-Aware Personalized Federated Learning in Heterogeneous Environmentscited by 0