mlsanityData to 7 Oct 2026

Dynamic Gradient Aggregation for Federated Domain Adaptation

arXiv 2106.07578 ↗First posted 14 Jun 2021cs.LG cs.DCcited by 4

Not on the map. Why?

Abstract (arXiv)

In this paper, a new learning algorithm for Federated Learning (FL) is introduced. The proposed scheme is based on a weighted gradient aggregation using two-step optimization to offer a flexible training pipeline. Herein, two different flavors of the aggregation method are presented, leading to an order of magnitude improvement in convergence speed compared to other distributed or FL training algorithms like BMUF and FedAvg. Further, the aggregation algorithm acts as a regularizer of the gradient quality. We investigate the effect of our FL algorithm in supervised and unsupervised Speech Recognition (SR) scenarios. The experimental validation is performed based on three tasks: first, the LibriSpeech task showing a speed-up of 7x and 6% word error rate reduction (WERR) compared to the baseline results. The second task is based on session adaptation providing 20% WERR over a powerful LAS model. Finally, our unsupervised pipeline is applied to the conversational SR task. The proposed FL system outperforms the baseline systems in both convergence speed and overall model performance.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 16 of its 36 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

16 of 36 references.

References · 16

  1. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  2. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,313
  3. 2020Adaptive Federated Optimizationcited by 1,036
  4. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
  5. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  6. 2018Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselinescited by 228
  7. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  8. 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 193
  9. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
  10. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 131
  11. 2018Federated Learning for Keyword Spottingcited by 95
  12. 2017Online Deep Learning: Learning Deep Neural Networks on the Flycited by 86
4 more
  1. 2015Variance Reduction in SGD by Distributed Importance Samplingcited by 79
  2. 2019Speech Recognition with Augmented Synthesized Speechcited by 60
  3. 2015Online Learning to Samplecited by 9
  4. 2020Federated Transfer Learning with Dynamic Gradient Aggregationcited by 8

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 1.

4 papers.

  1. 2022FLUTE: A Scalable, Extensible Framework for High-Performance Federated Learning Simulationscited by 27
  2. 2022Federated Multilingual Models for Medical Transcript Analysiscited by 11
  3. 2022UniFed: All-In-One Federated Learning Platform to Unify Open-Source Frameworkscited by 3
  4. 2023Project Florida: Federated Learning Made Easycited by 2