mlsanityData to 14 Oct 2026

Distributed Personalized Empirical Risk Minimization

arXiv 2310.17761 ↗First posted 26 Oct 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper advocates a new paradigm Personalized Empirical Risk Minimization (PERM) to facilitate learning from heterogeneous data sources without imposing stringent constraints on computational resources shared by participating devices. In PERM, we aim to learn a distinct model for each client by learning who to learn with and personalizing the aggregation of local empirical losses by effectively estimating the statistical discrepancy among data distributions, which entails optimal statistical accuracy for all local distributions and overcomes the data heterogeneity issue. To learn personalized models at scale, we propose a distributed algorithm that replaces the standard model averaging with model shuffling to simultaneously optimize PERM objectives for all devices. This also allows us to learn distinct model architectures (e.g., neural networks with different numbers of parameters) for different clients, thus confining underlying memory and compute resources of individual clients. We rigorously analyze the convergence of the proposed algorithm and conduct experiments that corroborate the effectiveness of the proposed paradigm.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 47 of its 63 references that are in the map. First posted on arXiv 26 Oct 2023.

Builds on

47 of 63 references.

References · 47

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  5. 2018Federated Learning with Non-IID Datacited by 1,108
  6. 2018LEAF: A Benchmark for Federated Settingscited by 792
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  8. 2017Federated Multi-Task Learningcited by 672
  9. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  10. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  11. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  12. 2019Agnostic Federated Learningcited by 456
35 more
  1. 2020Adaptive Personalized Federated Learningcited by 358
  2. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  3. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
  4. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  5. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 270
  6. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  7. 2021FedKD: Communication Efficient Federated Learning via Knowledge Distillationcited by 187
  8. 2019SCAFFOLD: Stochastic Controlled Averaging for Federated Learningcited by 175
  9. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 163
  10. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  11. 2019On the Convergence of Local Descent Methods in Federated Learningcited by 153
  12. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  13. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  14. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  15. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  16. 2019Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronizationcited by 107
  17. 2021Distributionally Robust Federated Averagingcited by 104
  18. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  19. 2020Local SGD: Unified Theory and New Efficient Methodscited by 88
  20. 2020Federated Accelerated Stochastic Gradient Descentcited by 85
  21. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  22. 2020SGD with shuffling: optimal rates without component convexity and large epoch requirementscited by 58
  23. 2020A Unified Single-loop Alternating Gradient Projection Algorithm for Nonconvex-Concave and Convex-Nonconcave Minimax Problemscited by 57
  24. 2023FedSpeed: Larger Local Interval, Less Communication Round, and Higher Generalization Accuracycited by 49
  25. 2022A Unified Analysis of Federated Learning with Arbitrary Client Participationcited by 45
  26. 2021Anarchic Federated Learningcited by 42
  27. 2019Semi-Cyclic Stochastic Gradient Descentcited by 40
  28. 2022On the Unreasonable Effectiveness of Federated Averaging with Heterogeneous Datacited by 38
  29. 2018Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimizationcited by 34
  30. 2021Personalized Federated Learning: A Unified Framework and Universal Optimization Techniquescited by 34
  31. 2022On the Convergence of Clustered Federated Learningcited by 31
  32. 2023On the Convergence of Federated Averaging with Cyclic Client Participationcited by 31
  33. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  34. 2020A No-Free-Lunch Theorem for MultiTask Learningcited by 28
  35. 2019Robust Learning from Untrusted Sourcescited by 27

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 4.

6 papers.

  1. 2025Tackling Feature and Sample Heterogeneity in Decentralized Multi-Task Learning: A Sheaf-Theoretic Approachcited by 10
  2. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 5
  3. 2023f-FERM: A Scalable Framework for Robust Fair Empirical Risk Minimizationcited by 4
  4. 2024AdaTrans: Feature-wise and Sample-wise Adaptive Transfer Learning for High-dimensional Regressioncited by 3
  5. 2025RCCDA: Adaptive Model Updates in the Presence of Concept Drift under a Constrained Resource Budgetcited by 2
  6. 2025On the Convergence and Stability of Distributed Sub-model Trainingcited by 1