# Private Language Model Adaptation for Speech Recognition

arXiv: https://arxiv.org/abs/2110.10026 · HTML: https://mlsanity.com/paper/2110.10026/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 28 Sep 2021.

## Builds on (most cited first, 17 of 17)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Federated Optimization: Distributed Machine Learning for On-Device Intelligence](https://mlsanity.com/paper/1610.02527/) (2016)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Federated Meta-Learning with Fast Convergence and Efficient Communication](https://mlsanity.com/paper/1802.07876/) (2018)
- [Character-Aware Neural Language Models](https://mlsanity.com/paper/1508.06615/) (2015)
- [An analysis of incorporating an external language model into a sequence-to-sequence model](https://mlsanity.com/paper/1712.01996/) (2017)
- [Language Modeling with Deep Transformers](https://mlsanity.com/paper/1905.04226/) (2019)
- [Federated Learning for Keyword Spotting](https://mlsanity.com/paper/1810.05512/) (2018)
- [Learning Private Neural Language Modeling with Attentive Aggregation](https://mlsanity.com/paper/1812.07108/) (2018)
- [Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework](https://mlsanity.com/paper/2010.15965/) (2020)
- [Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer](https://mlsanity.com/paper/2010.13878/) (2020)
- [Federated Acoustic Modeling For Automatic Speech Recognition](https://mlsanity.com/paper/2102.04429/) (2021)
- [A Parallelizable Lattice Rescoring Strategy with Neural Language Models](https://mlsanity.com/paper/2103.05081/) (2021)

## Built on by (most cited first, 4 of 4)

- [Unveiling the Potential of LLM-Based ASR on Chinese Open-Source Datasets](https://mlsanity.com/paper/2405.02132/) (2024)
- [Recovering from Privacy-Preserving Masking with Large Language Models](https://mlsanity.com/paper/2309.08628/) (2023)
- [Adaptive Multi-Corpora Language Model Training for Speech Recognition](https://mlsanity.com/paper/2211.05121/) (2022)
- [Correction Focused Language Model Training for Speech Recognition](https://mlsanity.com/paper/2310.11003/) (2023)

## Abstract (arXiv)

Speech model adaptation is crucial to handle the discrepancy between server-side proxy training data and actual data received on local devices of users. With the use of federated learning (FL), we introduce an efficient approach on continuously adapting neural network language models (NNLMs) on private devices with applications on automatic speech recognition (ASR). To address the potential speech transcription errors in the on-device training corpus, we perform empirical studies on comparing various strategies of leveraging token-level confidence scores to improve the NNLM quality in the FL settings. Experiments show that compared with no model adaptation, the proposed method achieves relative 2.6% and 10.8% word error rate (WER) reductions on two speech evaluation datasets, respectively. We also provide analysis in evaluating privacy guarantees of our presented procedure.