mlsanityData to 14 Oct 2026

FedSkip: Combatting Statistical Heterogeneity with Federated Skip Aggregation

arXiv 2212.07224 ↗First posted 14 Dec 2022cs.LG cs.DCcited by 16

Not on the map. Why?

Abstract (arXiv)

The statistical heterogeneity of the non-independent and identically distributed (non-IID) data in local clients significantly limits the performance of federated learning. Previous attempts like FedProx, SCAFFOLD, MOON, FedNova and FedDyn resort to an optimization perspective, which requires an auxiliary term or re-weights local updates to calibrate the learning bias or the objective inconsistency. However, in addition to previous explorations for improvement in federated averaging, our analysis shows that another critical bottleneck is the poorer optima of client models in more heterogeneous conditions. We thus introduce a data-driven approach called FedSkip to improve the client optima by periodically skipping federated averaging and scattering local models to the cross devices. We provide theoretical analysis of the possible benefit from FedSkip and conduct extensive experiments on a range of datasets to demonstrate that FedSkip achieves much higher accuracy, better aggregation efficiency and competing communication efficiency. Source code is available at: https://github.com/MediaBrain-SJTU/FedSkip.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 14 Dec 2022.

Builds on

28 of 37 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  6. 2018Federated Learning with Non-IID Datacited by 1,108
  7. 2019Deep Leakage from Gradientscited by 1,021
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  9. 2018LEAF: A Benchmark for Federated Settingscited by 792
  10. 2021Model-Contrastive Federated Learningcited by 766
  11. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  12. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
16 more
  1. 2020Federated Learning with Matched Averagingcited by 564
  2. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  3. 2019Fair Resource Allocation in Federated Learningcited by 394
  4. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  5. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
  6. 2019A Survey on Distributed Machine Learningcited by 188
  7. 2020Federated Visual Classification with Real-World Data Distributioncited by 128
  8. 2020FedVision: An Online Visual Object Detection Platform Powered by Federated Learningcited by 124
  9. 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 82
  10. 2021Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learningcited by 78
  11. 2019Federated Transfer Reinforcement Learning for Autonomous Drivingcited by 44
  12. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 39
  13. 2021Device-Cloud Collaborative Learning for Recommendationcited by 34
  14. 2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
  15. 2020Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimizationcited by 26
  16. 2022Device-Cloud Collaborative Recommendation via Meta Controllercited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 8; 2025: 2; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  2. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 67
  3. 2024Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimizationcited by 28
  4. 2024HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learningcited by 14
  5. 2024Diversified Batch Selection for Training Accelerationcited by 14
  6. 2024Federated Learning with Bilateral Curation for Partially Class-Disjoint Datacited by 13
  7. 2024Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapterscited by 12
  8. 2023Combating Representation Learning Disparity with Geometric Harmonizationcited by 11
  9. 2023MAS: Towards Resource-Efficient Federated Multiple-Task Learningcited by 9
  10. 2024Federated Learning under Partially Class-Disjoint Data via Manifold Reshapingcited by 4
  11. 2024A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needscited by 2
  12. 2024Neighborhood and Global Perturbations Supported SAM in Federated Learning: From Local Tweaks To Global Awarenesscited by 0
4 more
  1. 2025Distributionally Robust Federated Learning: An ADMM Algorithmcited by 0
  2. 2025Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learningcited by 0
  3. 2026FedFFT: Taming Client Drift in Federated SAM via Spectral Perturbation Filteringcited by 0
  4. 2026SpecGradFilter: A Spectral Gradient Filtering Framework for Taming Federated Heterogeneitycited by 0