mlsanityData to 7 Oct 2026

Supernet Training for Federated Image Classification under System Heterogeneity

arXiv 2206.01366 ↗First posted 3 Jun 2022cs.LG cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Efficient deployment of deep neural networks across many devices and resource constraints, particularly on edge devices, is one of the most challenging problems in the presence of data-privacy preservation issues. Conventional approaches have evolved to either improve a single global model while keeping each local heterogeneous training data decentralized (i.e. data heterogeneity; Federated Learning (FL)) or to train an overarching network that supports diverse architectural settings to address heterogeneous systems equipped with different computational capabilities (i.e. system heterogeneity; Neural Architecture Search). However, few studies have considered both directions simultaneously. This paper proposes the federation of supernet training (FedSup) framework to consider both scenarios simultaneously, i.e., where clients send and receive a supernet that contains all possible architectures sampled from itself. The approach is inspired by observing that averaging parameters during model aggregation for FL is similar to weight-sharing in supernet training. Thus, the proposed FedSup framework combines a weight-sharing approach widely used for training single shot models with FL averaging (FedAvg). Furthermore, we develop an efficient algorithm (E-FedSup) by sending the sub-model to clients on the broadcast stage to reduce communication costs and training overhead, including several strategies to enhance supernet training in the FL environment. We verify the proposed approach with extensive empirical evaluations. The resulting framework also ensures data and model heterogeneity robustness on several standard benchmarks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 67 of its 81 references that are in the map. First posted on arXiv 3 Jun 2022.

Builds on

67 of 81 references.

References · 67

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  6. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  9. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  10. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
38 more
  1. 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,097
  2. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  3. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  4. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 778
  5. 2021Model-Contrastive Federated Learningcited by 762
  6. 2017Dynamic Routing Between Capsulescited by 689
  7. 2021MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classificationcited by 682
  8. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  9. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 666
  10. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  11. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  12. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  13. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  14. 2019Agnostic Federated Learningcited by 451
  15. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  16. 2021Dynamic Neural Networks: A Surveycited by 405
  17. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  18. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
  19. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  20. 2017SkipNet: Learning Dynamic Routing in Convolutional Networkscited by 328
  21. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  22. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  23. 2018Slimmable Neural Networkscited by 301
  24. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  25. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  26. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  27. 2020Federated learning with hierarchical clustering of local updates to improve training on non-IID datacited by 222
  28. 2019Universally Slimmable Networks and Improved Training Techniquescited by 219
  29. 2021Personalized Federated Learning using Hypernetworkscited by 210
  30. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  31. 2021Evaluating Gradient Inversion Attacks and Defenses in Federated Learningcited by 196
  32. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  33. 2020BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Modelscited by 187
  34. 2017BlockDrop: Dynamic Inference Paths in Residual Networkscited by 185
  35. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  36. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  37. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 174
  38. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2024SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overheadcited by 9
  2. 2023Navigating Data Heterogeneity in Federated Learning A Semi-Supervised Federated Object Detectioncited by 4
  3. 2024Towards Privacy-Preserving Medical Imaging: Federated Learning with Differential Privacy and Secure Aggregation Using a Modified ResNet Architecturecited by 1
  4. 2024AdaS&S: a One-Shot Supernet Approach for Automatic Embedding Size Search in Deep Recommender Systemcited by 0