# Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout

arXiv: https://arxiv.org/abs/2210.16105 · HTML: https://mlsanity.com/paper/2210.16105/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 54 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 30 of 36)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients](https://mlsanity.com/paper/2010.01264/) (2020)
- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [Asynchronous Federated Optimization](https://mlsanity.com/paper/1903.03934/) (2019)
- [Federated Learning for Internet of Things: A Comprehensive Survey](https://mlsanity.com/paper/2104.07914/) (2021)
- [Sparse Communication for Distributed Gradient Descent](https://mlsanity.com/paper/1704.05021/) (2017)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Federated Learning with Buffered Asynchronous Aggregation](https://mlsanity.com/paper/2106.06639/) (2021)
- [Revisiting Distributed Synchronous SGD](https://mlsanity.com/paper/1702.05800/) (2017)
- [Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation](https://mlsanity.com/paper/1903.07424/) (2019)
- [Asynchronous Federated Learning on Heterogeneous Devices: A Survey](https://mlsanity.com/paper/2109.04269/) (2021)
- [Papaya: Practical, Private, and Scalable Federated Learning](https://mlsanity.com/paper/2111.04877/) (2021)
- [Learning Ordered Representations with Nested Dropout](https://mlsanity.com/paper/1402.0915/) (2014)
- [SAFA: a Semi-Asynchronous Protocol for Fast Federated Learning with Low Overhead](https://mlsanity.com/paper/1910.01355/) (2019)
- [Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning](https://mlsanity.com/paper/2206.08307/) (2022)
- [An Asynchronous Parallel Stochastic Coordinate Descent Algorithm](https://mlsanity.com/paper/1311.1873/) (2013)
- [Decentralized Federated Learning for UAV Networks: Architecture, Challenges, and Opportunities](https://mlsanity.com/paper/2104.07557/) (2021)
- [Distributed Learning of Deep Neural Networks using Independent Subnet Training](https://mlsanity.com/paper/1910.02120/) (2019)
- [Double Quantization for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1805.10111/) (2018)
- [ResIST: Layer-Wise Decomposition of ResNets for Distributed Training](https://mlsanity.com/paper/2107.00961/) (2021)
- [Multi-Sample Dropout for Accelerated Training and Better Generalization](https://mlsanity.com/paper/1905.09788/) (2019)

## Built on by (most cited first, 12 of 12)

- [Asynchronous Federated Reinforcement Learning with Policy Gradient Updates: Algorithm Design and Convergence Analysis](https://mlsanity.com/paper/2404.08003/) (2024)
- [FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneity](https://mlsanity.com/paper/2404.09816/) (2024)
- [Towards a Better Theoretical Understanding of Independent Subnetwork Training](https://mlsanity.com/paper/2306.16484/) (2023)
- [Momentum Approximation in Asynchronous Private Federated Learning](https://mlsanity.com/paper/2402.09247/) (2024)
- [Review of Mathematical Optimization in Federated Learning](https://mlsanity.com/paper/2412.01630/) (2024)
- [Generative Artificial Intelligence for Internet of Things Computing: A Systematic Survey](https://mlsanity.com/paper/2504.07635/) (2025)
- [TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training](https://mlsanity.com/paper/2511.03983/) (2025)
- [FedAgg: Adaptive Federated Learning with Aggregated Gradients](https://mlsanity.com/paper/2303.15799/) (2023)
- [Traversal Learning: A Lossless And Efficient Distributed Learning Framework](https://mlsanity.com/paper/2504.07471/) (2025)
- [Federated Learning in Offline and Online EMG Decoding: A Privacy and Performance Perspective](https://mlsanity.com/paper/2507.12652/) (2025)
- [Health+: Empowering Individuals via Unifying Health Data](https://mlsanity.com/paper/2602.19319/) (2026)
- [A Comparative Study of Federated Learning Aggregation Strategies under Homogeneous and Heterogeneous Data Distributions](https://mlsanity.com/paper/2605.11010/) (2026)

## Abstract (arXiv)

Asynchronous learning protocols have regained attention lately, especially in the Federated Learning (FL) setup, where slower clients can severely impede the learning process. Herein, we propose textttAsyncDrop, a novel asynchronous FL framework that utilizes dropout regularization to handle device heterogeneity in distributed settings. Overall, textttAsyncDrop achieves better performance compared to state of the art asynchronous methodologies, while resulting in less communication and training time overheads. The key idea revolves around creating ``submodels'' out of the global model, and distributing their training to workers, based on device heterogeneity. We rigorously justify that such an approach can be theoretically characterized. We implement our approach and compare it against other asynchronous baselines, both by design and by adapting existing synchronous FL algorithms to asynchronous scenarios. Empirically, textttAsyncDrop reduces the communication cost and training time, while matching or improving the final test accuracy in diverse non-i.i.d. FL scenarios.