mlsanityData to 7 Oct 2026

Exploiting Shared Representations for Personalized Federated Learning

arXiv 2102.07078 ↗First posted 14 Feb 2021cs.LG math.OCcited by 537

Not on the map. Why?

Abstract (arXiv)

Deep neural networks have shown the ability to extract universal feature representations from data such as images and text that have been useful for a variety of learning tasks. However, the fruits of representation learning have yet to be fully-realized in federated settings. Although data in federated settings is often non-i.i.d. across clients, the success of centralized deep learning suggests that data often shares a global feature representation, while the statistical heterogeneity across clients or tasks is concentrated in the labels. Based on this intuition, we propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. We prove that this method obtains linear convergence to the ground-truth representation with near-optimal sample complexity in a linear setting, demonstrating that it can efficiently reduce the problem dimension for each client. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions, for example in meta-learning and multi-task learning. Further, extensive experimental results show the empirical improvement of our method over alternative personalized federated learning approaches in federated environments with heterogeneous data.

Key facts. Cited by 537 papers in the map (data through 7 Oct 2026). Builds on 34 of its 54 references that are in the map. First posted on arXiv 14 Feb 2021.

Builds on

34 of 54 references.

References · 34

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  3. 2020Adaptive Federated Optimizationcited by 1,036
  4. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  5. 2018LEAF: A Benchmark for Federated Settingscited by 780
  6. 2017Federated Multi-Task Learningcited by 669
  7. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  8. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  9. 2019Federated Learning with Personalization Layerscited by 533
  10. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  11. 2020Adaptive Personalized Federated Learningcited by 353
  12. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
22 more
  1. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  2. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  3. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  4. 2020Few-Shot Learning via Learning the Representation, Provablycited by 213
  5. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  6. 2015The Benefit of Multitask Representation Learningcited by 194
  7. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  8. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  9. 2020Provable Meta-Learning of Linear Representationscited by 161
  10. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  11. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 159
  12. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  13. 2019Federated Evaluation of On-device Personalizationcited by 147
  14. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  15. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  16. 2015Low-rank Solutions of Linear Matrix Equations via Procrustes Flowcited by 97
  17. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  18. 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
  19. 2020Meta-learning for mixed linear regressioncited by 44
  20. 2016Convergence Analysis for Rectangular Matrix Completion Using Burer-Monteiro Factorization and Gradient Descentcited by 37
  21. 2014Efficient Representations for Life-Long Learning and Autoencodingcited by 23
  22. 2016Finding Low-Rank Solutions via Non-Convex Matrix Factorization, Efficiently and Provablycited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 68; 2023: 113; 2024: 144; 2025: 103; 2026: 94.

537 papers.

Sort:

By area:

  1. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  2. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  3. 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
  4. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 141
  5. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 121
  6. 2022Federated Learning with Partial Model Personalizationcited by 114
  7. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  8. 2024Selective Aggregation for Low-Rank Adaptation in Federated Learningcited by 80
  9. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  10. 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 66
  11. 2021Personalized Federated Learning through Local Memorizationcited by 57
  12. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
38 more
  1. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  2. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  3. 2023Efficient Model Personalization in Federated Learning via Client-Specific Prompt Generationcited by 54
  4. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 54
  5. 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
  6. 2024Global and Local Prompts Cooperation via Optimal Transport for Federated Learningcited by 45
  7. 2022Federated Learning via Inexact ADMMcited by 44
  8. 2022pFL-Bench: A Comprehensive Benchmark for Personalized Federated Learningcited by 44
  9. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  10. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  11. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 38
  12. 2023Eliminating Domain Bias for Federated Learning in Representation Spacecited by 38
  13. 2022FedTP: Federated Learning by Transformer Personalizationcited by 36
  14. 2023FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policycited by 36
  15. 2021Specificity-Preserving Federated Learning for MR Image Reconstructioncited by 35
  16. 2022FedPop: A Bayesian Approach for Personalised Federated Learningcited by 35
  17. 2022On the Convergence of Clustered Federated Learningcited by 31
  18. 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 30
  19. 2022Architecture Agnostic Federated Learning for Neural Networkscited by 30
  20. 2023Towards Federated Foundation Models: Scalable Dataset Pipelines for Group-Structured Learningcited by 29
  21. 2022Personalizing Federated Medical Image Segmentation via Local Calibrationcited by 28
  22. 2022FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learningcited by 27
  23. 2023Learning from Similar Linear Representations: Adaptivity, Minimaxity, and Robustnesscited by 27
  24. 2024FedSelect: Personalized Federated Learning with Customized Selection of Parameters for Fine-Tuningcited by 27
  25. 2022FiT: Parameter Efficient Few-shot Transfer Learning for Personalized and Federated Image Classificationcited by 26
  26. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  27. 2024Harmonizing Generalization and Personalization in Federated Prompt Learningcited by 26
  28. 2024Time-FFM: Towards LM-Empowered Federated Foundation Model for Time Series Forecastingcited by 26
  29. 2021Sample Efficient Linear Meta-Learning by Alternating Minimizationcited by 25
  30. 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
  31. 2023Federated Temporal Difference Learning with Linear Function Approximation under Environmental Heterogeneitycited by 25
  32. 2024AdaFGL: A New Paradigm for Federated Node Classification with Topology Heterogeneitycited by 25
  33. 2022FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentationcited by 24
  34. 2021Towards Federated Learning on Time-Evolving Heterogeneous Datacited by 23
  35. 2023Bold but Cautious: Unlocking the Potential of Personalized Federated Learning through Cautiously Aggressive Collaborationcited by 23
  36. 2023FS-Real: Towards Real-World Cross-Device Federated Learningcited by 22
  37. 2022IOP-FL: Inside-Outside Personalization for Federated Medical Image Segmentationcited by 21
  38. 2023PeFLL: Personalized Federated Learning by Learning to Learncited by 21

50 most cited of 537 All 537 (JSON)