mlsanityData to 7 Oct 2026

SPATL: Salient Parameter Aggregation and Transfer Learning for Heterogeneous Clients in Federated Learning

arXiv 2111.14345 ↗First posted 29 Nov 2021cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Federated learning~(FL) facilitates the training and deploying AI models on edge devices. Preserving user data privacy in FL introduces several challenges, including expensive communication costs, limited resources, and data heterogeneity. In this paper, we propose SPATL, an FL method that addresses these issues by: (a) introducing a salient parameter selection agent and communicating selected parameters only; (b) splitting a model into a shared encoder and a local predictor, and transferring its knowledge to heterogeneous clients via the locally customized predictor. Additionally, we leverage a gradient control mechanism to further speed up model convergence and increase robustness of training processes. Experiments demonstrate that SPATL reduces communication overhead, accelerates model inference, and enables stable training processes with better results compared to state-of-the-art methods. Our approach reduces communication cost by up to 86.45%, accelerates local inference by reducing up to 39.7% FLOPs on VGG-11, and requires 7.4 × less communication overhead when training ResNet-20.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 54 of its 72 references that are in the map. First posted on arXiv 29 Nov 2021.

Builds on

54 of 72 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  8. 2018Federated Learning with Non-IID Datacited by 1,099
  9. 2020Adaptive Federated Optimizationcited by 1,036
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  12. 2018LEAF: A Benchmark for Federated Settingscited by 780
38 more
  1. 2021Model-Contrastive Federated Learningcited by 762
  2. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  3. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  4. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 449
  5. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 407
  6. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  7. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  8. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 365
  9. 2018Client Selection for Federated Learning with Heterogeneous Resources in Mobile Edgecited by 347
  10. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  11. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  12. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
  13. 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 284
  14. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  15. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 247
  16. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  17. 2021Personalized Federated Learning using Hypernetworkscited by 210
  18. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  19. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  20. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  21. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 185
  22. 2019Privacy-preserving Federated Brain Tumour Segmentationcited by 169
  23. 2018Multi-Institutional Deep Learning Modeling Without Sharing Patient Data: A Feasibility Study on Brain Tumor Segmentationcited by 148
  24. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 122
  25. 2020APQ: Joint Search for Network Architecture, Pruning and Quantization Policycited by 98
  26. 2020Provable Defense against Privacy Leakage in Federated Learning from Representation Perspectivecited by 94
  27. 2021Federated Evaluation and Tuning for On-Device Personalization: System Design & Applicationscited by 84
  28. 2020Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approachcited by 80
  29. 2020EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruningcited by 80
  30. 2021Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learningcited by 78
  31. 2019Towards Efficient Model Compression via Learned Global Rankingcited by 75
  32. 2021Convolutional Neural Network Pruning with Structural Redundancy Reductioncited by 73
  33. 2020FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tierscited by 64
  34. 2020IBM Federated Learning: an Enterprise Framework White Paper V0.1cited by 57
  35. 2019Federated Learning Of Out-Of-Vocabulary Wordscited by 51
  36. 2020DSA: More Efficient Budgeted Pruning via Differentiable Sparsity Allocationcited by 47
  37. 2021Cluster-driven Graph Federated Learning over Multiple Domainscited by 43
  38. 2018Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implicationscited by 35

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 5; 2024: 4.

11 papers.

  1. 2023Federated Foundation Models: Privacy-Preserving and Collaborative Learning for Large Modelscited by 49
  2. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 12
  3. 2023Bridging the Gap Between Foundation Models and Heterogeneous Federated Learningcited by 9
  4. 2024FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharingcited by 8
  5. 2024A Comprehensive Survey of Federated Transfer Learning: Challenges, Methods and Applicationscited by 8
  6. 2023TraceFL: Interpretability-Driven Debugging in Federated Learning via Neuron Provenancecited by 1
  7. 2023LEFL: Low Entropy Client Sampling in Federated Learningcited by 1
  8. 2024DynamicFL: Federated Learning with Dynamic Communication Resource Allocationcited by 1
  9. 2022Resource-Aware Heterogeneous Federated Learning using Neural Architecture Searchcited by 0
  10. 2022Addressing Data Heterogeneity in Decentralized Learning via Topological Pre-processingcited by 0
  11. 2023Accelerating Domain-aware Deep Learning Models with Distributed Trainingcited by 0