FedNet2Net: Saving Communication and Computations in Federated Learning with Model Growing
Not on the map. Why?
Abstract (arXiv)
Federated learning (FL) is a recently developed area of machine learning, in which the private data of a large number of distributed clients is used to develop a global model under the coordination of a central server without explicitly exposing the data. The standard FL strategy has a number of significant bottlenecks including large communication requirements and high impact on the clients' resources. Several strategies have been described in the literature trying to address these issues. In this paper, a novel scheme based on the notion of "model growing" is proposed. Initially, the server deploys a small model of low complexity, which is trained to capture the data complexity during the initial set of rounds. When the performance of such a model saturates, the server switches to a larger model with the help of function-preserving transformations. The model complexity increases as more data is processed by the clients, and the overall process continues until the desired performance is achieved. Therefore, the most complex model is broadcast only at the final stage in our approach resulting in substantial reduction in communication cost and client computational requirements. The proposed approach is tested extensively on three standard benchmarks and is shown to achieve substantial reduction in communication and client computation while achieving comparable accuracy when compared to the current most effective strategies.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 19 Jul 2022.
Builds on
21 of 33 references.
References · 21
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
- 2014How transferable are features in deep neural networks?cited by 1,216
- 2018LEAF: A Benchmark for Federated Settingscited by 780
- 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
- 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
- 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 245
- 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
- 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
- 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 178
9 more
- 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
- 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
- 2019Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networkscited by 79
- 2020Ternary Compression for Communication-Efficient Federated Learningcited by 57
- 2020Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learningcited by 37
- 2020Communication-Efficient Federated Distillationcited by 28
- 2021ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Trainingcited by 27
- 2020Secure Aggregation with Heterogeneous Quantization in Federated Learningcited by 21
- 2021Communication-Efficient Federated Learning with Dual-Side Low-Rank Compressioncited by 10
Built on by
Citing papers per year: 2023: 1.
1 paper.