mlsanityData to 7 Oct 2026

Linear Speedup in Personalized Collaborative Learning

arXiv 2111.05968 ↗First posted 10 Nov 2021cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Collaborative training can improve the accuracy of a model for a user by trading off the model's bias (introduced by using data from other users who are potentially different) against its variance (due to the limited amount of data on any single user). In this work, we formalize the personalized collaborative learning problem as a stochastic optimization of a task 0 while giving access to N related but different tasks 1,..., N. We provide convergence guarantees for two algorithms in this setting -- a popular collaboration method known as weighted gradient averaging, and a novel bias correction method -- and explore conditions under which we can achieve linear speedup w.r.t. the number of auxiliary tasks N. Further, we also empirically study their performance confirming our theoretical insights.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 42 of its 53 references that are in the map. First posted on arXiv 10 Nov 2021.

Builds on

42 of 53 references.

References · 42

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2019Advances and Open Problems in Federated Learningcited by 3,021
  3. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  5. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  6. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  7. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  8. 2019Agnostic Federated Learningcited by 451
  9. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  10. 2019Fair Resource Allocation in Federated Learningcited by 389
  11. 2020Adaptive Personalized Federated Learningcited by 353
  12. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
30 more
  1. 2021A Field Guide to Federated Optimizationcited by 312
  2. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  3. 2019Momentum-Based Variance Reduction in Non-Convex SGDcited by 302
  4. 2019A Little Is Enough: Circumventing Defenses For Distributed Learningcited by 285
  5. 2020Federated Learning of a Mixture of Global and Local Modelscited by 208
  6. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  7. 2015The Benefit of Multitask Representation Learningcited by 194
  8. 2020On the Theory of Transfer Learning: The Importance of Task Diversitycited by 186
  9. 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
  10. 2013Minimizing Finite Sums with the Stochastic Average Gradientcited by 164
  11. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  12. 2019Federated Evaluation of On-device Personalizationcited by 147
  13. 2020Survey of Personalization Techniques for Federated Learningcited by 144
  14. 2020Salvaging Federated Learning by Local Adaptationcited by 142
  15. 2020Learning from History for Byzantine Robust Optimizationcited by 137
  16. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 131
  17. 2019Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Modelcited by 116
  18. 2020On the Convergence of SGD with Biased Gradientscited by 79
  19. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  20. 2020Model-sharing Games: Analyzing Federated Learning Under Voluntary Participationcited by 60
  21. 2020Auxiliary Learning by Implicit Differentiationcited by 45
  22. 2016AIDE: Fast and Communication Efficient Distributed Optimizationcited by 42
  23. 2021Personalized Federated Learning: A Unified Framework and Universal Optimization Techniquescited by 34
  24. 2021Sample Efficient Linear Meta-Learning by Alternating Minimizationcited by 25
  25. 2020Auxiliary Task Reweighting for Minimum-data Learningcited by 23
  26. 2017Byzantine-Tolerant Machine Learningcited by 18
  27. 2021WAFFLE: Weighted Averaging for Personalized Federated Learningcited by 9
  28. 2021Optimal Model Averaging: Towards Personalized Collaborative Learningcited by 7
  29. 2021Provable Adaptation across Multiway Domains via Representation Learningcited by 3
  30. 2021User-Centric Federated Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 2; 2025: 2.

8 papers.

  1. 2022Optimization with Access to Auxiliary Informationcited by 12
  2. 2023Incentivizing Honesty among Competitors in Collaborative Learning and Optimizationcited by 6
  3. 2023DePRL: Achieving Linear Convergence Speedup in Personalized Decentralized Learning with Shared Representationscited by 5
  4. 2024Incentivizing Truthful Collaboration in Heterogeneous Federated Learningcited by 3
  5. 2024Scalable Decentralized Algorithms for Online Personalized Mean Estimationcited by 2
  6. 2025Adaptive collaboration for online personalized distributed learning with heterogeneous clientscited by 2
  7. 2025Personalized Collaborative Learning with Affinity-Based Variance Reductioncited by 1
  8. 2023Utilizing Free Clients in Federated Learning for Focused Model Enhancementcited by 0