Straggler-resilient Federated Learning: Tackling Computation Heterogeneity with Layer-wise Partial Model Training in Mobile Edge Network
Not on the map. Why?
Abstract (arXiv)
Federated Learning (FL) enables many resource-limited devices to train a model collaboratively without data sharing. However, many existing works focus on model-homogeneous FL, where the global and local models are the same size, ignoring the inherently heterogeneous computational capabilities of different devices and restricting resource-constrained devices from contributing to FL. In this paper, we consider model-heterogeneous FL and propose Federated Partial Model Training (FedPMT), where devices with smaller computational capabilities work on partial models (subsets of the global model) and contribute to the global model. Different from Dropout-based partial model generation, which removes neurons in hidden layers at random, model training in FedPMT is achieved from the back-propagation perspective. As such, all devices in FedPMT prioritize the most crucial parts of the global model. Theoretical analysis shows that the proposed partial model training design has a similar convergence rate to the widely adopted Federated Averaging (FedAvg) algorithm, O(1/T), with the sub-optimality gap enlarged by a constant factor related to the model splitting design in FedPMT. Empirical results show that FedPMT significantly outperforms the existing benchmark FedDrop. Meanwhile, compared to the popular model-homogeneous benchmark, FedAvg, FedPMT reaches the learning target in a shorter completion time, thus achieving a better trade-off between learning accuracy and completion time.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 16 Nov 2023.
Builds on
45 of 62 references.
References · 45
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
- 2018Local SGD Converges Fast and Communicates Littlecited by 595
- 2020An Efficient Framework for Clustered Federated Learningcited by 505
- 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 408
- 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 391
33 more
- 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
- 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
- 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
- 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
- 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
- 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 306
- 2018Slimmable Neural Networkscited by 301
- 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
- 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 222
- 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
- 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
- 2018Federated Learning via Over-the-Air Computationcited by 215
- 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
- 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
- 2019On the Convergence of Local Descent Methods in Federated Learningcited by 150
- 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
- 2019Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocationcited by 93
- 2018Wireless Network Intelligence at the Edgecited by 90
- 2014Convolutional Neural Networks at Constrained Time Costcited by 81
- 2020Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approachcited by 80
- 2020Convergence of Update Aware Device Scheduling for Federated Learning at the Wireless Edgecited by 71
- 2020Fast-Convergent Federated Learning with Adaptive Weightingcited by 67
- 2021Tackling System and Statistical Heterogeneity for Federated Learning with Adaptive Client Samplingcited by 66
- 2020Cost-Effective Federated Learning Designcited by 61
- 2021Federated Dropout -- A Simple Approach for Enabling Federated Learning on Resource Constrained Devicescited by 56
- 2020Design and Analysis of Uplink and Downlink Communications for Federated Learningcited by 48
- 2021Node Selection Toward Faster Convergence for Federated Learning on Non-IID Datacited by 39
- 2017meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfittingcited by 34
- 2023Federated Learning-Empowered AI-Generated Content in Wireless Networkscited by 26
- 2019Distributed Learning of Deep Neural Networks using Independent Subnet Trainingcited by 20
- 2021Masked Training of Neural Networks with Partial Gradientscited by 15
- 2021Joint Superposition Coding and Training for Federated Learning over Multi-Width Neural Networkscited by 14
- 2023Split Learning for Distributed Collaborative Training of Deep Learning Models in Health Informaticscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.