mlsanityData to 7 Oct 2026

End-to-End Speech Recognition from Federated Acoustic Models

arXiv 2104.14297 ↗First posted 29 Apr 2021cs.SD cs.LG eess.AScited by 30

Not on the map. Why?

Abstract (arXiv)

Training Automatic Speech Recognition (ASR) models under federated learning (FL) settings has attracted a lot of attention recently. However, the FL scenarios often presented in the literature are artificial and fail to capture the complexity of real FL systems. In this paper, we construct a challenging and realistic ASR federated experimental setup consisting of clients with heterogeneous data distributions using the French and Italian sets of the CommonVoice dataset, a large heterogeneous dataset containing thousands of different speakers, acoustic environments and noises. We present the first empirical study on attention-based sequence-to-sequence End-to-End (E2E) ASR model with three aggregation weighting strategies -- standard FedAvg, loss-based aggregation and a novel word error rate (WER)-based aggregation, compared in two realistic FL scenarios: cross-silo with 10 clients and cross-device with 2K and 4K clients. Our analysis on E2E ASR from heterogeneous and realistic federated acoustic models provides the foundations for future research and development of realistic FL-based ASR applications.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 29 Apr 2021.

Builds on

29 of 44 references.

References · 29

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  5. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,358
  7. 2018Federated Learning with Non-IID Datacited by 1,099
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 449
  10. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  11. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  12. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
17 more
  1. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  2. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  3. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  4. 2019pyannote.audio: neural building blocks for speaker diarizationcited by 175
  5. 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
  6. 2018Federated Learning for Keyword Spottingcited by 95
  7. 2018Pre-training on high-resource speech recognition improves low-resource speech-to-text translationcited by 70
  8. 2020Can Federated Learning Save The Planet?cited by 52
  9. 2020Training Speech Recognition Models with Federated Learning: A Quality/Cost Frameworkcited by 46
  10. 2019Transformers with convolutional context for ASRcited by 44
  11. 2017Exploring Neural Transducers for End-to-End Speech Recognitioncited by 43
  12. 2020Training Keyword Spotting Models on Non-IID Data with Federated Learningcited by 36
  13. 2016Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognitioncited by 33
  14. 2019Meta Learning for End-to-End Low-Resource Speech Recognitioncited by 29
  15. 2020Improving on-device speaker verification using federated learning with privacycited by 29
  16. 2021Federated Acoustic Modeling For Automatic Speech Recognitioncited by 20
  17. 2016TristouNet: Triplet Loss for Speaker Turn Embeddingcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 7; 2024: 7; 2025: 3; 2026: 4.

30 papers.

Sort:

By area:

  1. 2024Automatic Speech Recognition using Advanced Deep Learning Approaches: A surveycited by 50
  2. 2023L-DAWA: Layer-wise Divergence Aware Weight Aggregation in Federated Self-Supervised Visual Representation Learningcited by 29
  3. 2023Federated Learning for Data Streamscited by 15
  4. 2022Federated Self-supervised Learning for Video Understandingcited by 13
  5. 2022Federated Self-supervised Speech Representations: Are We There Yet?cited by 10
  6. 2023Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASRcited by 8
  7. 2024Communication-Efficient Personalized Federated Learning for Speech-to-Text Taskscited by 8
  8. 2023Federated Learning for ASR based on Wav2vec 2.0cited by 7
  9. 2022Federated Domain Adaptation for ASR with Full Self-Supervisioncited by 6
  10. 2022Decoupled Federated Learning for ASR with Non-IID Datacited by 6
  11. 2021Federated Self-Training for Semi-Supervised Audio Recognitioncited by 5
  12. 2022Federated Self-Supervised Learning for Acoustic Event Classificationcited by 5
18 more
  1. 2022FedNST: Federated Noisy Student Training for Automatic Speech Recognitioncited by 5
  2. 2022Protea: Client Profiling within Federated Systems using Flowercited by 4
  3. 2022ILASR: Privacy-Preserving Incremental Learning for Automatic Speech Recognition at Production Scalecited by 4
  4. 2024Improved Modelling of Federated Datasets using Mixtures-of-Dirichlet-Multinomialscited by 3
  5. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  6. 2023Cross-device Federated Learning for Mobile Health Diagnostics: A First Study on COVID-19 Detectioncited by 2
  7. 2024Exploring Federated Self-Supervised Learning for General Purpose Audio Understandingcited by 2
  8. 2024FedAnchor: Enhancing Federated Semi-Supervised Learning with Label Contrastive Loss for Unlabeled Clientscited by 2
  9. 2024Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clientscited by 1
  10. 2025Regularized Federated Learning for Privacy-Preserving Dysarthric and Elderly Speech Recognitioncited by 1
  11. 2026Federated Heterogeneous Language Model Optimization for Hybrid Automatic Speech Recognitioncited by 1
  12. 2023Ed-Fed: A generic federated learning framework with resource-aware client selection for edge devicescited by 0
  13. 2023Federated Representation Learning for Automatic Speech Recognitioncited by 0
  14. 2024FedRepOpt: Gradient Re-parametrized Optimizers in Federated Learningcited by 0
  15. 2025FedSCA: Federated Tuning with Similarity-guided Collaborative Aggregation for Heterogeneous Medical Image Segmentationcited by 0
  16. 2026Towards Personalized Federated Learning for Dysarthric Speech Recognitioncited by 0
  17. 2026SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studiescited by 0
  18. 2026Federated Multilingual Speech-LLMs: Architecture and Aggregation Strategy Benchmarkingcited by 0