# Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework

arXiv: https://arxiv.org/abs/2010.15965 · HTML: https://mlsanity.com/paper/2010.15965/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2020.

## Built on by (most cited first, 30 of 46)

- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learning](https://mlsanity.com/paper/2206.03655/) (2022)
- [Federated Learning Meets Natural Language Processing: A Survey](https://mlsanity.com/paper/2107.12603/) (2021)
- [Backdoor Attacks and Defenses in Federated Learning: Survey, Challenges and Future Research Directions](https://mlsanity.com/paper/2303.02213/) (2023)
- [End-to-End Speech Recognition from Federated Acoustic Models](https://mlsanity.com/paper/2104.14297/) (2021)
- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Auto-weighted Robust Federated Learning with Corrupted Data Sources](https://mlsanity.com/paper/2101.05880/) (2021)
- [Enabling On-Device Training of Speech Recognition Models with Federated Dropout](https://mlsanity.com/paper/2110.03634/) (2021)
- [Training Machine Learning models at the Edge: A Survey](https://mlsanity.com/paper/2403.02619/) (2024)
- [Privacy attacks for automatic speech recognition acoustic models in a federated learning framework](https://mlsanity.com/paper/2111.03777/) (2021)
- [A Method to Reveal Speaker Identity in Distributed ASR Training, and How to Counter It](https://mlsanity.com/paper/2104.07815/) (2021)
- [A Hybrid Architecture for Federated and Centralized Learning](https://mlsanity.com/paper/2105.03288/) (2021)
- [Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data](https://mlsanity.com/paper/2105.04727/) (2021)
- [AdaSplit: Adaptive Trade-offs for Resource-constrained Distributed Deep Learning](https://mlsanity.com/paper/2112.01637/) (2021)
- [Federated Pruning: Improving Neural Network Efficiency with Federated Learning](https://mlsanity.com/paper/2209.06359/) (2022)
- [Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASR](https://mlsanity.com/paper/2309.13102/) (2023)
- [Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks](https://mlsanity.com/paper/2401.10070/) (2024)
- [Federated Domain Adaptation for ASR with Full Self-Supervision](https://mlsanity.com/paper/2203.15966/) (2022)
- [Decoupled Federated Learning for ASR with Non-IID Data](https://mlsanity.com/paper/2206.09102/) (2022)
- [Federated Self-Supervised Learning for Acoustic Event Classification](https://mlsanity.com/paper/2203.11997/) (2022)
- [FedNST: Federated Noisy Student Training for Automatic Speech Recognition](https://mlsanity.com/paper/2206.02797/) (2022)
- [Personalized Federated Learning on Long-Tailed Data via Adversarial Feature Augmentation](https://mlsanity.com/paper/2303.15168/) (2023)
- [Zero-Shot Federated Learning with New Classes for Audio Classification](https://mlsanity.com/paper/2106.10019/) (2021)
- [Private Language Model Adaptation for Speech Recognition](https://mlsanity.com/paper/2110.10026/) (2021)
- [Personalized Federated Learning with Exact Stochastic Gradient Descent](https://mlsanity.com/paper/2202.09848/) (2022)
- [Avoid Overfitting User Specific Information in Federated Keyword Spotting](https://mlsanity.com/paper/2206.08864/) (2022)
- [ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scale](https://mlsanity.com/paper/2207.09078/) (2022)
- [Analyzing Inference Privacy Risks Through Gradients in Machine Learning](https://mlsanity.com/paper/2408.16913/) (2024)
- [Incremental Layer-wise Self-Supervised Learning for Efficient Speech Domain Adaptation On Device](https://mlsanity.com/paper/2110.00155/) (2021)
- [Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decoding](https://mlsanity.com/paper/2202.05209/) (2022)

## Abstract (arXiv)

We propose using federated learning, a decentralized on-device learning paradigm, to train speech recognition models. By performing epochs of training on a per-user basis, federated learning must incur the cost of dealing with non-IID data distributions, which are expected to negatively affect the quality of the trained model. We propose a framework by which the degree of non-IID-ness can be varied, consequently illustrating a trade-off between model quality and the computational cost of federated training, which we capture through a novel metric. Finally, we demonstrate that hyper-parameter optimization and appropriate use of variational noise are sufficient to compensate for the quality impact of non-IID distributions, while decreasing the cost.