mlsanityData to 7 Oct 2026

Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneity

arXiv 2305.18285 ↗First posted 29 May 2023cs.LG cs.AI math.OCcited by 8

Not on the map. Why?

Abstract (arXiv)

We present a partially personalized formulation of Federated Learning (FL) that strikes a balance between the flexibility of personalization and cooperativeness of global training. In our framework, we split the variables into global parameters, which are shared across all clients, and individual local parameters, which are kept private. We prove that under the right split of parameters, it is possible to find global parameters that allow each client to fit their data perfectly, and refer to the obtained problem as overpersonalized. For instance, the shared global parameters can be used to learn good data representations, whereas the personalized layers are fine-tuned for a specific client. Moreover, we present a simple algorithm for the partially personalized formulation that offers significant benefits to all clients. In particular, it breaks the curse of data heterogeneity in several settings, such as training with local steps, asynchronous training, and Byzantine-robust training.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 49 of its 73 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

49 of 73 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  10. 2018Byzantine-Robust Distributed Learning: Towards Optimal Statistical Ratescited by 698
  11. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  12. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
37 more
  1. 2019Federated Learning with Personalization Layerscited by 533
  2. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 516
  3. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  4. 2018The Hidden Vulnerability of Distributed Learning in Byzantiumcited by 324
  5. 2019Robust Aggregation for Federated Learningcited by 314
  6. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  7. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 285
  8. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  9. 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 264
  10. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  11. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  12. 2020Privacy and Robustness in Federated Learning: Attacks and Defensescited by 163
  13. 2018Federated Meta-Learning with Fast Convergence and Efficient Communicationcited by 154
  14. 2017The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learningcited by 150
  15. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  16. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  17. 2020Learning from History for Byzantine Robust Optimizationcited by 137
  18. 2019Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulationcited by 136
  19. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  20. 2020Federated Visual Classification with Real-World Data Distributioncited by 126
  21. 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 125
  22. 2022Federated Learning with Partial Model Personalizationcited by 114
  23. 2018Byzantine Stochastic Gradient Descentcited by 81
  24. 2019First Analysis of Local GD on Heterogeneous Datacited by 80
  25. 2019Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attackscited by 79
  26. 2016Learning in games with continuous action sets and unknown payoff functionscited by 78
  27. 2021Federated Reconstruction: Partially Local Federated Learningcited by 78
  28. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  29. 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
  30. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  31. 2022FedNest: Federated Bilevel, Minimax, and Compositional Optimizationcited by 60
  32. 2020Byzantine-Resilient Non-Convex Stochastic Gradient Descentcited by 56
  33. 2021Stochastic Gradient Descent-Ascent and Consensus Optimization for Smooth Games: Convergence Analysis under Expected Co-coercivitycited by 43
  34. 2021Optimistic Dual Extrapolation for Coherent Non-monotone Variational Inequalitiescited by 29
  35. 2022Can 5th Generation Local Training Methods Support Client Sampling? Yes!cited by 22
  36. 2022Local Stochastic Bilevel Optimization with Momentum-Based Variance Reductioncited by 20
  37. 2021Secure Distributed Training at Scalecited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 2; 2026: 1.

8 papers.

  1. 2024Fine-Tuning Personalization in Federated Learning to Mitigate Adversarial Clientscited by 5
  2. 2024Towards Faster Decentralized Stochastic Optimization with Communication Compressioncited by 4
  3. 2024Incentivizing Truthful Collaboration in Heterogeneous Federated Learningcited by 3
  4. 2025Bant: Byzantine Antidote via Trial Function and Trust Scorescited by 3
  5. 2023Adaptive Model Pruning and Personalization for Federated Learning over Wireless Networkscited by 2
  6. 2024Collaborative and Efficient Personalization with Mixtures of Adaptorscited by 1
  7. 2025Personalized Federated Training of Diffusion Models with Privacy Guaranteescited by 0
  8. 2026Collaborative and Efficient Fine-tuning: Leveraging Task Similaritycited by 0