mlsanityData to 14 Oct 2026

Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout

arXiv 2210.16105 ↗First posted 28 Oct 2022cs.LG cs.AI cs.ITcited by 12

Not on the map. Why?

Abstract (arXiv)

Asynchronous learning protocols have regained attention lately, especially in the Federated Learning (FL) setup, where slower clients can severely impede the learning process. Herein, we propose textttAsyncDrop, a novel asynchronous FL framework that utilizes dropout regularization to handle device heterogeneity in distributed settings. Overall, textttAsyncDrop achieves better performance compared to state of the art asynchronous methodologies, while resulting in less communication and training time overheads. The key idea revolves around creating ``submodels'' out of the global model, and distributing their training to workers, based on device heterogeneity. We rigorously justify that such an approach can be theoretically characterized. We implement our approach and compare it against other asynchronous baselines, both by design and by adapting existing synchronous FL algorithms to asynchronous scenarios. Empirically, textttAsyncDrop reduces the communication cost and training time, while matching or improving the final test accuracy in diverse non-i.i.d. FL scenarios.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 28 Oct 2022.

Builds on

36 of 54 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2016Deep Networks with Stochastic Depthcited by 979
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  7. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  8. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
  10. 2015Variational Dropout and the Local Reparameterization Trickcited by 490
  11. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  12. 2021A Field Guide to Federated Optimizationcited by 315
24 more
  1. 2019Asynchronous Federated Optimizationcited by 267
  2. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 252
  3. 2017Sparse Communication for Distributed Gradient Descentcited by 249
  4. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  5. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 190
  6. 2017Revisiting Distributed Synchronous SGDcited by 142
  7. 2019Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregationcited by 100
  8. 2021Asynchronous Federated Learning on Heterogeneous Devices: A Surveycited by 93
  9. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 90
  10. 2014Learning Ordered Representations with Nested Dropoutcited by 84
  11. 2019SAFA: a Semi-Asynchronous Protocol for Fast Federated Learning with Low Overheadcited by 82
  12. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 79
  13. 2013An Asynchronous Parallel Stochastic Coordinate Descent Algorithmcited by 26
  14. 2021Decentralized Federated Learning for UAV Networks: Architecture, Challenges, and Opportunitiescited by 22
  15. 2019Distributed Learning of Deep Neural Networks using Independent Subnet Trainingcited by 20
  16. 2018Double Quantization for Communication-Efficient Distributed Optimizationcited by 19
  17. 2021ResIST: Layer-Wise Decomposition of ResNets for Distributed Trainingcited by 18
  18. 2019Multi-Sample Dropout for Accelerated Training and Better Generalizationcited by 14
  19. 2021Stragglers Are Not Disaster: A Hybrid Federated Learning Algorithm with Delayed Gradientscited by 13
  20. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  21. 2021GIST: Distributed Training for Large-Scale Graph Convolutional Networkscited by 10
  22. 2022Applied Federated Learning: Architectural Design for Robust and Efficient Learning in Privacy Aware Settingscited by 7
  23. 2015PASSCoDe: Parallel ASynchronous Stochastic dual Co-ordinate Descentcited by 4
  24. 2013NOMAD: Non-locking, stOchastic Multi-machine algorithm for Asynchronous and Decentralized matrix completioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 4; 2026: 2.

12 papers.

  1. 2024Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysiscited by 21
  2. 2024FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneitycited by 13
  3. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  4. 2024Momentum Approximation in Asynchronous Private Federated Learningcited by 3
  5. 2024Review of Mathematical Optimization in Federated Learningcited by 3
  6. 2025Generative Artificial Intelligence for Internet of Things Computing: A Systematic Surveycited by 3
  7. 2025TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Trainingcited by 1
  8. 2023FedAgg: Adaptive Federated Learning with Aggregated Gradientscited by 0
  9. 2025Traversal Learning: A Lossless And Efficient Distributed Learning Frameworkcited by 0
  10. 2025Federated Learning in Offline and Online EMG Decoding: A Privacy and Performance Perspectivecited by 0
  11. 2026Health+: Empowering Individuals via Unifying Health Datacited by 0
  12. 2026A Comparative Study of Federated Learning Aggregation Strategies under Homogeneous and Heterogeneous Data Distributionscited by 0