mlsanityData to 7 Oct 2026

FedAvg with Fine Tuning: Local Updates Lead to Representation Learning

arXiv 2205.13692 ↗First posted 27 May 2022cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

The Federated Averaging (FedAvg) algorithm, which consists of alternating between a few local stochastic gradient updates at client nodes, followed by a model averaging update at the server, is perhaps the most commonly used method in Federated Learning. Notwithstanding its simplicity, several empirical studies have illustrated that the output model of FedAvg, after a few fine-tuning steps, leads to a model that generalizes well to new unseen tasks. This surprising performance of such a simple method, however, is not fully understood from a theoretical point of view. In this paper, we formally investigate this phenomenon in the multi-task linear representation setting. We show that the reason behind generalizability of the FedAvg's output is its power in learning the common data representation among the clients' tasks, by leveraging the diversity among client data distributions via local updates. We formally establish the iteration complexity required by the clients for proving such result in the setting where the underlying shared representation is a linear map. To the best of our knowledge, this is the first such result for any setting. We also provide empirical evidence demonstrating FedAvg's representation learning ability in federated image classification with heterogeneous data.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 62 of its 82 references that are in the map. First posted on arXiv 27 May 2022.

Builds on

62 of 82 references.

References · 62

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  7. 2020Adaptive Federated Optimizationcited by 1,036
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  9. 2021Model-Contrastive Federated Learningcited by 762
  10. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  11. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  12. 2018Local SGD Converges Fast and Communicates Littlecited by 595
38 more
  1. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  2. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  3. 2019Federated Learning with Personalization Layerscited by 533
  4. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  5. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  6. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  7. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  8. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  9. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  10. 2019SGD: General Analysis and Improved Ratescited by 206
  11. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  12. 2015The Benefit of Multitask Representation Learningcited by 194
  13. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  14. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  15. 2020Provable Meta-Learning of Linear Representationscited by 161
  16. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 159
  17. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  18. 2020Is Local SGD Better than Minibatch SGD?cited by 151
  19. 2019Federated Evaluation of On-device Personalizationcited by 147
  20. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  21. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  22. 2020Optimal Client Sampling for Federated Learningcited by 123
  23. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  24. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  25. 2018Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithmscited by 106
  26. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 106
  27. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  28. 2020Federated Unsupervised Representation Learningcited by 92
  29. 2017On the convergence properties of a K-step averaging stochastic gradient descent algorithm for nonconvex optimizationcited by 91
  30. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  31. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  32. 2013Distributed optimization over time-varying directed graphscited by 77
  33. 2020From Local SGD to Local Fixed-Point Methods for Federated Learningcited by 74
  34. 2021Collaborative Unsupervised Visual Representation Learning from Decentralized Datacited by 68
  35. 2021Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learningcited by 52
  36. 2020Robust Meta-learning for Mixed Linear Regression with Small Batchescited by 41
  37. 2021How Fine-Tuning Allows for Effective Meta-Learningcited by 41
  38. 2020Communication-Efficient Federated Learning via Optimal Client Samplingcited by 39

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 17; 2024: 14; 2025: 10; 2026: 6.

55 papers.

Sort:

By area:

  1. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  2. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  3. 2023Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challengescited by 39
  4. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  5. 2022Learning to Generate Image Embeddings with User-level Differential Privacycited by 23
  6. 2025Secure On-Device Video OOD Detection Without Backpropagationcited by 19
  7. 2022GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexitycited by 17
  8. 2025Robust Federated Finetuning of LLMs via Alternating Optimization of LoRAcited by 17
  9. 2023Federated TD Learning over Finite-Rate Erasure Channels: Linear Speedup under Markovian Samplingcited by 14
  10. 2024MH-pFLID: Model Heterogeneous personalized Federated Learning via Injection and Distillation for Medical Data Analysiscited by 14
  11. 2023Share Your Representation Only: Guaranteed Improvement of the Privacy-Utility Tradeoff in Federated Learningcited by 13
  12. 2024Unlock the Potential of Large Language Models for Predictive Tabular Tasks in Data Science with Table-Specific Pretrainingcited by 12
38 more
  1. 2022Straggler-Resilient Personalized Federated Learningcited by 9
  2. 2023Profit: Benchmarking Personalization and Robustness Trade-off in Federated Prompt Tuningcited by 9
  3. 2022On the effectiveness of partial variance reduction in federated learning with heterogeneous datacited by 8
  4. 2023Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Datacited by 7
  5. 2022Flow: Per-Instance Personalized Federated Learning Through Dynamic Routingcited by 6
  6. 2023Fake It Till Make It: Federated Learning with Consensus-Oriented Generationcited by 6
  7. 2023Personalised Federated Learning On Heterogeneous Feature Spacescited by 5
  8. 2024Improving Local Training in Federated Learning via Temperature Scalingcited by 5
  9. 2024Backpropagation-Free Multi-modal On-Device Model Adaptation via Cloud-Device Collaborationcited by 5
  10. 2024Adaptive Aggregation Weights for Federated Segmentation of Pancreas MRIcited by 5
  11. 2023Leveraging Function Space Aggregation for Federated Learning at Scalecited by 4
  12. 2024Differentiated Aggregation to Improve Generalization in Federated Learningcited by 4
  13. 2022Quantifying the Impact of Label Noise on Federated Learningcited by 3
  14. 2023A review of federated learning in renewable energy applications: Potential, challenges, and future directionscited by 3
  15. 2024Wind turbine condition monitoring based on intra- and inter-farm federated learningcited by 3
  16. 2022Towards Fleet-wide Sharing of Wind Turbine Condition Information through Privacy-preserving Federated Learningcited by 2
  17. 2023Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneitycited by 2
  18. 2024Overcoming label shift with target-aware federated learningcited by 2
  19. 2024IPMN Risk Assessment under Federated Learning Paradigmcited by 2
  20. 2025Collaborative Split Federated Learning with Parallel Training and Aggregationcited by 2
  21. 2025Federated Learning for Large Models in Medical Imaging: A Comprehensive Reviewcited by 2
  22. 2022Federated Representation Learning via Maximal Coding Rate Reductioncited by 1
  23. 2023Federated Neural Compression Under Heterogeneous Datacited by 1
  24. 2024Client-supervised Federated Learning: Towards One-model-for-all Personalizationcited by 1
  25. 2024FedSat: A Statistical Aggregation Approach for Class Imbalanced Clients in Federated Learningcited by 1
  26. 2024Sequential Federated Learning in Hierarchical Architecture on Non-IID Datasetscited by 1
  27. 2025FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraintscited by 1
  28. 2023Federated Learning on Patient Data for Privacy-Protecting Polycystic Ovary Syndrome Treatmentcited by 0
  29. 2023Federated Deep Equilibrium Learning: Harnessing Compact Global Representations to Enhance Personalizationcited by 0
  30. 2023Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Datacited by 0
  31. 2024Personalized Interpretation on Federated Learning: A Virtual Concepts approachcited by 0
  32. 2024FedCAP: Robust Federated Learning via Customized Aggregation and Personalizationcited by 0
  33. 2025Aggregating Low Rank Adapters in Federated Fine-tuningcited by 0
  34. 2025Meta-learning of shared linear representations beyond well-specified linear regressioncited by 0
  35. 2025Trade-off in Estimating the Number of Byzantine Clients in Federated Learningcited by 0
  36. 2025FedMGP: Personalized Federated Learning with Multi-Group Text-Visual Promptscited by 0
  37. 2025A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learningcited by 0
  38. 2026Collaborative and Efficient Fine-tuning: Leveraging Task Similaritycited by 0

50 most cited of 55 All 55 (JSON)