Adaptive collaboration for online personalized distributed learning with heterogeneous clients
Not on the map. Why?
Abstract (arXiv)
We study the problem of online personalized decentralized learning with N statistically heterogeneous clients collaborating to accelerate local training. An important challenge in this setting is to select relevant collaborators to reduce gradient variance while mitigating the introduced bias. To tackle this, we introduce a gradient-based collaboration criterion, allowing each client to dynamically select peers with similar gradients during the optimization process. Our criterion is motivated by a refined and more general theoretical analysis of the All-for-one algorithm, proved to be optimal in Even et al. (2022) for an oracle collaboration scheme. We derive excess loss upper-bounds for smooth objective functions, being either strongly convex, non-convex, or satisfying the Polyak-Lojasiewicz condition; our analysis reveals that the algorithm acts as a variance reduction method where the speed-up depends on a sufficient variance. We put forward two collaboration methods instantiating the proposed general schema; and we show that one variant preserves the optimality of All-for-one. We validate our results with experiments on synthetic and real datasets.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 39 references that are in the map. First posted on arXiv 9 Jul 2025.
Builds on
28 of 39 references.
References · 28
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2018Federated Optimization in Heterogeneous Networkscited by 2,987
- 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
- 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
- 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
- 2020Personalized Federated Learning with Moreau Envelopescited by 618
- 2019Federated Learning with Personalization Layerscited by 541
- 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
- 2020An Efficient Framework for Clustered Federated Learningcited by 511
- 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
- 2021Towards Personalized Federated Learningcited by 464
16 more
- 2020Adaptive Personalized Federated Learningcited by 358
- 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
- 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
- 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 192
- 2020Survey of Personalization Techniques for Federated Learningcited by 145
- 2020Salvaging Federated Learning by Local Adaptationcited by 142
- 2022FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settingscited by 125
- 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 123
- 2022Mechanisms that Incentivize Data Sharing in Federated Learningcited by 39
- 2023Provably Personalized and Robust Federated Learningcited by 11
- 2021WAFFLE: Weighted Averaging for Personalized Federated Learningcited by 9
- 2021Linear Speedup in Personalized Collaborative Learningcited by 9
- 2023FedDWA: Personalized Federated Learning with Dynamic Weight Adjustmentcited by 9
- 2024Unravelling in Collaborative Learningcited by 6
- 2024CoBo: Collaborative Learning via Bilevel Optimizationcited by 5
- 2024In-depth Analysis of Low-rank Matrix Factorisation in a Federated Settingcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.