mlsanityData to 14 Oct 2026

On the Importance and Applicability of Pre-Training for Federated Learning

arXiv 2206.11488 ↗First posted 23 Jun 2022cs.LG cs.AI cs.CVcited by 71

Not on the map. Why?

Abstract (arXiv)

Pre-training is prevalent in nowadays deep learning to improve the learned model's performance. However, in the literature on federated learning (FL), neural networks are mostly initialized with random weights. These attract our interest in conducting a systematic study to explore pre-training for FL. Across multiple visual recognition benchmarks, we found that pre-training can not only improve FL, but also close its accuracy gap to the counterpart centralized learning, especially in the challenging cases of non-IID clients' data. To make our findings applicable to situations where pre-trained models are not directly available, we explore pre-training with synthetic data or even with clients' data in a decentralized manner, and found that they can already improve FL notably. Interestingly, many of the techniques we explore are complementary to each other to further boost the performance, and we view this as a critical result toward scaling up deep FL for real-world applications. We conclude our paper with an attempt to understand the effect of pre-training on FL. We found that pre-training enables the learned global models under different clients' data conditions to converge to the same loss basin, and makes global aggregation in FL more stable. Nevertheless, pre-training seems to not alleviate local model drifting, a fundamental problem in FL under non-IID data.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Builds on 77 of its 100 references that are in the map. First posted on arXiv 23 Jun 2022.

Builds on

77 of 100 references.

References · 77

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  11. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  12. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  3. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
  4. 2019Advances and Open Problems in Federated Learningcited by 3,046
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  9. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  10. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  13. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  14. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  15. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  16. 2018Federated Learning with Non-IID Datacited by 1,108
  17. 2020Adaptive Federated Optimizationcited by 1,046
  18. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  19. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  20. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  21. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  22. 2018LEAF: A Benchmark for Federated Settingscited by 792
  23. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  24. 2021Model-Contrastive Federated Learningcited by 766
  25. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  26. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  27. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  28. 2018Do Better ImageNet Models Transfer Better?cited by 565
  29. 2020Federated Learning with Matched Averagingcited by 564
  30. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  31. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  32. 2018Rethinking ImageNet Pre-trainingcited by 377
  33. 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
  34. 2020What is being transferred in transfer learning?cited by 368
  35. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  36. 2020Rethinking the Value of Labels for Improving Class-Imbalanced Learningcited by 216
  37. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  38. 2021Self-supervised Pretraining of Visual Features in the Wildcited by 198

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 24; 2024: 28; 2025: 13; 2026: 6.

71 papers.

Sort:

By area:

  1. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  2. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  3. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  4. 2023Integration of Large Language Models and Federated Learningcited by 31
  5. 2024FedLLM-Bench: Realistic Benchmarks for Federated Learning of Large Language Modelscited by 30
  6. 2023Asynchronous Federated Continual Learningcited by 29
  7. 2024Enhancing One-Shot Federated Learning Through Data and Ensemble Co-Boostingcited by 27
  8. 2024Federated Large Language Models: Current Progress and Future Directionscited by 27
  9. 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
  10. 2023GPT-FL: Generative Pre-trained Model-Assisted Federated Learningcited by 24
  11. 2023Layer-wise Linear Mode Connectivitycited by 22
  12. 2024From Challenges and Pitfalls to Recommendations and Opportunities: Implementing Federated Learning in Healthcarecited by 16
38 more
  1. 2024FedFisher: Leveraging Fisher Information for One-Shot Federated Learningcited by 15
  2. 2024animal2vec and MeerKAT: A self-supervised transformer for rare-event raw audio input and a large-scale reference dataset for bioacousticscited by 14
  3. 2023Guiding The Last Layer in Federated Learning with Pre-Trained Modelscited by 13
  4. 2023On the Efficacy of Differentially Private Few-shot Image Classificationcited by 12
  5. 2024Towards Multi-modal Transformers in Federated Learningcited by 12
  6. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  7. 2024Recurrent Early Exits for Federated Learning with Heterogeneous Clientscited by 12
  8. 2024Federated Online Adaptation for Deep Stereocited by 12
  9. 2024Task Arithmetic Through The Lens Of One-Shot Federated Learningcited by 12
  10. 2024AFL: A Single-Round Analytic Approach for Federated Learning with Pre-trained Modelscited by 11
  11. 2023ZooPFL: Exploring Black-box Foundation Models for Personalized Federated Learningcited by 9
  12. 2023Making Batch Normalization Great in Federated Deep Learningcited by 8
  13. 2023FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learningcited by 8
  14. 2023NeFL: Nested Model Scaling for Federated Learning with System Heterogeneous Clientscited by 7
  15. 2023Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Datacited by 7
  16. 2025Resilient Contrastive Pre-training under Non-Stationary Driftcited by 7
  17. 2025Capture Global Feature Statistics for One-Shot Federated Learningcited by 6
  18. 2023Leveraging Foundation Models to Improve Lightweight Clients in Federated Learningcited by 5
  19. 2024SPD-CFL: Stepwise Parameter Dropout for Efficient Continual Federated Learningcited by 5
  20. 2023SelfFed: Self-Supervised Federated Learning for Data Heterogeneity and Label Scarcity in Medical Imagescited by 4
  21. 2023Federated Model Aggregation via Self-Supervised Priors for Highly Imbalanced Medical Image Classificationcited by 4
  22. 2024Federated Learning Client Pruning for Noisy Labelscited by 4
  23. 2024Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learningcited by 4
  24. 2023Where to Begin? From Random to Foundation Model Instructed Initialization in Federated Learning for Medical Image Segmentationcited by 3
  25. 2023Federated Learning Empowered by Generative Contentcited by 3
  26. 2024Federated Learning over Connected Modescited by 3
  27. 2024Unlocking the Potential of Model Calibration in Federated Learningcited by 3
  28. 2024FedBaF: Federated Learning Aggregation Biased by a Foundation Modelcited by 3
  29. 2025AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Datacited by 3
  30. 2023Federated Deep Learning for Intrusion Detection in IoT Networkscited by 2
  31. 2023Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneitycited by 2
  32. 2024FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learningcited by 2
  33. 2024FedPylot: Navigating Federated Learning for Real-Time Object Detection in Internet of Vehiclescited by 2
  34. 2024TriplePlay: Enhancing Federated Learning with CLIP for Non-IID Data and Resource Efficiencycited by 2
  35. 2025FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Modelscited by 2
  36. 2023ARIA: On the Interaction Between Architectures, Initialization and Aggregation Methods for Federated Visual Classificationcited by 1
  37. 2024Rethinking the Starting Point: Collaborative Pre-Training for Federated Downstream Taskscited by 1
  38. 2024Why pre-training is beneficial for downstream classification tasks?cited by 1

50 most cited of 71 All 71 (JSON)