# Federated Learning in ASR: Not as Easy as You Think

arXiv: https://arxiv.org/abs/2109.15108 · HTML: https://mlsanity.com/paper/2109.15108/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Sep 2021.

## Built on by (most cited first, 14 of 14)

- [A Review of Speech-centric Trustworthy Machine Learning: Privacy, Safety, and Fairness](https://mlsanity.com/paper/2212.09006/) (2022)
- [Training Machine Learning models at the Edge: A Survey](https://mlsanity.com/paper/2403.02619/) (2024)
- [Privacy attacks for automatic speech recognition acoustic models in a federated learning framework](https://mlsanity.com/paper/2111.03777/) (2021)
- [Federated Learning for ASR based on Wav2vec 2.0](https://mlsanity.com/paper/2302.10790/) (2023)
- [Federated Domain Adaptation for ASR with Full Self-Supervision](https://mlsanity.com/paper/2203.15966/) (2022)
- [Improving Automatic Speech Recognition for Non-Native English with Transfer Learning and Language Model Decoding](https://mlsanity.com/paper/2202.05209/) (2022)
- [Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization](https://mlsanity.com/paper/2501.06663/) (2025)
- [Retrieving Speaker Information from Personalized Acoustic Models for Speech Recognition](https://mlsanity.com/paper/2111.04194/) (2021)
- [Federated Learning in Healthcare: Model Misconducts, Security, Challenges, Applications, and Future Research Directions -- A Systematic Review](https://mlsanity.com/paper/2405.13832/) (2024)
- [Regularized Federated Learning for Privacy-Preserving Dysarthric and Elderly Speech Recognition](https://mlsanity.com/paper/2506.11069/) (2025)
- [Ed-Fed: A generic federated learning framework with resource-aware client selection for edge devices](https://mlsanity.com/paper/2307.07199/) (2023)
- [Federated Representation Learning for Automatic Speech Recognition](https://mlsanity.com/paper/2308.02013/) (2023)
- [Towards Personalized Federated Learning for Dysarthric Speech Recognition](https://mlsanity.com/paper/2606.13253/) (2026)
- [SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studies](https://mlsanity.com/paper/2607.25716/) (2026)

## Abstract (arXiv)

With the growing availability of smart devices and cloud services, personal speech assistance systems are increasingly used on a daily basis. Most devices redirect the voice recordings to a central server, which uses them for upgrading the recognizer model. This leads to major privacy concerns, since private data could be misused by the server or third parties. Federated learning is a decentralized optimization strategy that has been proposed to address such concerns. Utilizing this approach, private data is used for on-device training. Afterwards, updated model parameters are sent to the server to improve the global model, which is redistributed to the clients. In this work, we implement federated learning for speech recognition in a hybrid and an end-to-end model. We discuss the outcomes of these systems, which both show great similarities and only small improvements, pointing to a need for a deeper understanding of federated learning for speech recognition.