# Training Keyword Spotting Models on Non-IID Data with Federated Learning

arXiv: https://arxiv.org/abs/2005.10406 · HTML: https://mlsanity.com/paper/2005.10406/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 30 of 36)

- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Auditing Privacy Defenses in Federated Learning via Generative Gradient Leakage](https://mlsanity.com/paper/2203.15696/) (2022)
- [No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier](https://mlsanity.com/paper/2303.10058/) (2023)
- [Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning](https://mlsanity.com/paper/2103.05032/) (2021)
- [ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity](https://mlsanity.com/paper/2208.02507/) (2022)
- [On Convergence of FedProx: Local Dissimilarity Invariant Bounds, Non-smoothness and Beyond](https://mlsanity.com/paper/2206.05187/) (2022)
- [End-to-End Speech Recognition from Federated Acoustic Models](https://mlsanity.com/paper/2104.14297/) (2021)
- [Can Public Large Language Models Help Private Cross-device Federated Learning?](https://mlsanity.com/paper/2305.12132/) (2023)
- [Scaling Language Model Size in Cross-Device Federated Learning](https://mlsanity.com/paper/2204.09715/) (2022)
- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Enabling On-Device Training of Speech Recognition Models with Federated Dropout](https://mlsanity.com/paper/2110.03634/) (2021)
- [Federated Learning with Dynamic Transformer for Text to Speech](https://mlsanity.com/paper/2107.08795/) (2021)
- [Training Machine Learning models at the Edge: A Survey](https://mlsanity.com/paper/2403.02619/) (2024)
- [Privacy attacks for automatic speech recognition acoustic models in a federated learning framework](https://mlsanity.com/paper/2111.03777/) (2021)
- [Production federated keyword spotting via distillation, filtering, and joint federated-centralized training](https://mlsanity.com/paper/2204.06322/) (2022)
- [Analysis of Privacy Leakage in Federated Large Language Models](https://mlsanity.com/paper/2403.04784/) (2024)
- [A Method to Reveal Speaker Identity in Distributed ASR Training, and How to Counter It](https://mlsanity.com/paper/2104.07815/) (2021)
- [FLAME: Federated Learning Across Multi-device Environments](https://mlsanity.com/paper/2202.08922/) (2022)
- [Jointly Learning from Decentralized (Federated) and Centralized Data to Mitigate Distribution Shift](https://mlsanity.com/paper/2111.12150/) (2021)
- [Federated Learning for ASR based on Wav2vec 2.0](https://mlsanity.com/paper/2302.10790/) (2023)
- [Privacy-preserving Speech Emotion Recognition through Semi-Supervised Federated Learning](https://mlsanity.com/paper/2202.02611/) (2022)
- [Federated Self-Training for Semi-Supervised Audio Recognition](https://mlsanity.com/paper/2107.06877/) (2021)
- [On the Impact of Device and Behavioral Heterogeneity in Federated Learning](https://mlsanity.com/paper/2102.07500/) (2021)
- [Zero-Shot Federated Learning with New Classes for Audio Classification](https://mlsanity.com/paper/2106.10019/) (2021)
- [Avoid Overfitting User Specific Information in Federated Keyword Spotting](https://mlsanity.com/paper/2206.08864/) (2022)
- [Confidential Federated Computations](https://mlsanity.com/paper/2404.10764/) (2024)
- [Retrieving Speaker Information from Personalized Acoustic Models for Speech Recognition](https://mlsanity.com/paper/2111.04194/) (2021)
- [Federated Automatic Differentiation](https://mlsanity.com/paper/2301.07806/) (2023)
- [Efficient Language Model Architectures for Differentially Private Federated Learning](https://mlsanity.com/paper/2403.08100/) (2024)
- [Speech Privacy Leakage from Shared Gradients in Distributed Learning](https://mlsanity.com/paper/2302.10441/) (2023)

## Abstract (arXiv)

We demonstrate that a production-quality keyword-spotting model can be trained on-device using federated learning and achieve comparable false accept and false reject rates to a centrally-trained model. To overcome the algorithmic constraints associated with fitting on-device data (which are inherently non-independent and identically distributed), we conduct thorough empirical studies of optimization algorithms and hyperparameter configurations using large-scale federated simulations. To overcome resource constraints, we replace memory intensive MTR data augmentation with SpecAugment, which reduces the false reject rate by 56%. Finally, to label examples (given the zero visibility into on-device data), we explore teacher-student training.