mlsanityData to 14 Oct 2026

Importance of Smoothness Induced by Optimizers in FL4ASR: Towards Understanding Federated Learning for End-to-End ASR

arXiv 2309.13102 ↗First posted 22 Sep 2023eess.AS cs.DC cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

In this paper, we start by training End-to-End Automatic Speech Recognition (ASR) models using Federated Learning (FL) and examining the fundamental considerations that can be pivotal in minimizing the performance gap in terms of word error rate between models trained using FL versus their centralized counterpart. Specifically, we study the effect of (i) adaptive optimizers, (ii) loss characteristics via altering Connectionist Temporal Classification (CTC) weight, (iii) model initialization through seed start, (iv) carrying over modeling setup from experiences in centralized training to FL, e.g., pre-layer or post-layer normalization, and (v) FL-specific hyperparameters, such as number of local epochs, client sampling size, and learning rate scheduler, specifically for ASR under heterogeneous data distribution. We shed light on how some optimizers work better than others via inducing smoothness. We also summarize the applicability of algorithms, trends, and propose best practices from prior works in FL (in general) toward End-to-End ASR models.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 30 of its 40 references that are in the map. First posted on arXiv 22 Sep 2023.

Builds on

30 of 40 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  9. 2020Adaptive Federated Optimizationcited by 1,046
  10. 2019When Does Label Smoothing Help?cited by 988
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  12. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
18 more
  1. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  2. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  3. 2021Towards Personalized Federated Learningcited by 464
  4. 2015Attention-Based Models for Speech Recognitioncited by 401
  5. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  6. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  7. 2021WeNet: Production oriented Streaming and Non-streaming End-to-End Speech Recognition Toolkitcited by 125
  8. 2023Input Perturbation Reduces Exposure Bias in Diffusion Modelscited by 90
  9. 2020Contextual RNN-T For Open Domain ASRcited by 85
  10. 2020Training Speech Recognition Models with Federated Learning: A Quality/Cost Frameworkcited by 46
  11. 2021SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Trainingcited by 45
  12. 2021End-to-End Speech Recognition from Federated Acoustic Modelscited by 30
  13. 2018On Lipschitz Bounds of General Convolutional Neural Networkscited by 29
  14. 2021U2++: Unified Two-pass Bidirectional End-to-end Model for Speech Recognitioncited by 29
  15. 2022Recycling Model Updates in Federated Learning: Are Gradient Subspaces Low-Rank?cited by 16
  16. 2021Enabling On-Device Training of Speech Recognition Models with Federated Dropoutcited by 12
  17. 2022Addressing Heterogeneity in Federated Learning via Distributional Transformationcited by 6
  18. 2020Can we Generalize and Distribute Private Representation Learning?cited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 3.

8 papers.

  1. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  2. 2024Synergizing Foundation Models and Federated Learning: A Surveycited by 14
  3. 2024pfl-research: simulation framework for accelerating research in Private Federated Learningcited by 8
  4. 2024Federating Dynamic Models using Early-Exit Architectures for Automatic Speech Recognition on Heterogeneous Clientscited by 1
  5. 2025Regularized Federated Learning for Privacy-Preserving Dysarthric and Elderly Speech Recognitioncited by 1
  6. 2026Towards Personalized Federated Learning for Dysarthric Speech Recognitioncited by 0
  7. 2026SpeechLLM Meets Federated Learning for End-to-End ASR: English and Italian Case Studiescited by 0
  8. 2026A Practical Recipe for Semi-Supervised Federated ASR: Online Pseudo-Labels with Server Update Stabilizationcited by 0