# Reducing Impacts of System Heterogeneity in Federated Learning using Weight Update Magnitudes

arXiv: https://arxiv.org/abs/2208.14808 · HTML: https://mlsanity.com/paper/2208.14808/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 45 references that are in the map.
- First posted on arXiv 30 Aug 2022.

## Builds on (most cited first, 25 of 25)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [Think Locally, Act Globally: Federated Learning with Local and Global Representations](https://mlsanity.com/paper/2001.01523/) (2020)
- [Asynchronous Federated Optimization](https://mlsanity.com/paper/1903.03934/) (2019)
- [Model Pruning Enables Efficient Federated Learning on Edge Devices](https://mlsanity.com/paper/1909.12326/) (2019)
- [Applied Federated Learning: Improving Google Keyboard Query Suggestions](https://mlsanity.com/paper/1812.02903/) (2018)
- [Gradient Sparsification for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1710.09854/) (2017)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Federated Learning with Buffered Asynchronous Aggregation](https://mlsanity.com/paper/2106.06639/) (2021)
- [Expanding the Reach of Federated Learning by Reducing Client Resource Requirements](https://mlsanity.com/paper/1812.07210/) (2018)
- [Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach](https://mlsanity.com/paper/2001.04756/) (2020)
- [FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers](https://mlsanity.com/paper/2010.05958/) (2020)
- [FedAdapt: Adaptive Offloading for IoT Devices in Federated Learning](https://mlsanity.com/paper/2107.04271/) (2021)
- [Federated Learning Challenges and Opportunities: An Outlook](https://mlsanity.com/paper/2202.00807/) (2022)
- [FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients](https://mlsanity.com/paper/2201.11865/) (2022)
- [Gap Aware Mitigation of Gradient Staleness](https://mlsanity.com/paper/1909.10802/) (2019)
- [Taming Momentum in a Distributed Asynchronous Environment](https://mlsanity.com/paper/1907.11612/) (2019)
- [FedFly: Towards Migration in Edge-based Distributed Federated Learning](https://mlsanity.com/paper/2111.01516/) (2021)

## Abstract (arXiv)

The widespread adoption of handheld devices have fueled rapid growth in new applications. Several of these new applications employ machine learning models to train on user data that is typically private and sensitive. Federated Learning enables machine learning models to train locally on each handheld device while only synchronizing their neuron updates with a server. While this enables user privacy, technology scaling and software advancements have resulted in handheld devices with varying performance capabilities. This results in the training time of federated learning tasks to be dictated by a few low-performance straggler devices, essentially becoming a bottleneck to the entire training process. In this work, we aim to mitigate the performance bottleneck of federated learning by dynamically forming sub-models for stragglers based on their performance and accuracy feedback. To this end, we offer the Invariant Dropout, a dynamic technique that forms a sub-model based on the neuron update threshold. Invariant Dropout uses neuron updates from the non-straggler clients to develop a tailored sub-models for each straggler during each training iteration. All corresponding weights which have a magnitude less than the threshold are dropped for the iteration. We evaluate Invariant Dropout using five real-world mobile clients. Our evaluations show that Invariant Dropout obtains a maximum accuracy gain of 1.4% points over state-of-the-art Ordered Dropout while mitigating performance bottlenecks of stragglers.