Collaborative and Efficient Personalization with Mixtures of Adaptors
Not on the map. Why?
Abstract (arXiv)
Heterogenous data is prevalent in real-world federated learning. We propose a parameter-efficient framework, Federated Low-Rank Adaptive Learning (FLoRAL), that allows clients to personalize in groups by mixing between low-rank adaptors, where the mixtures are client-specific. FLoRAL is a model parameterization that casts personalized federated learning as a multi-task learning problem, with weight sharing as an implicit regularizer. It is memory-efficient, as the personalized parameters (i.e., base model + adaptors) are all federated. Our results show that FLoRAL can generalize better than a mixture of full models when data are scarce. It can also consistently personalize better than models with a locally tuned adaptor per client. This demonstrates the benefits of "federated personalization" and its robustness against overfitting. We derive the convergence rates and show theoretically that FLoRAL can lead to better variance reduction of the base model's gradients.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 4 Oct 2024.
Builds on
30 of 44 references.
References · 30
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
- 2018Local SGD Converges Fast and Communicates Littlecited by 595
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
- 2020An Efficient Framework for Clustered Federated Learningcited by 505
- 2018Sparsified SGD with Memorycited by 373
- 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
- 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
18 more
- 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
- 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
- 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 189
- 2022Federated Learning with Partial Model Personalizationcited by 114
- 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
- 2020Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descentcited by 70
- 2021Initialization and Regularization of Factorized Neural Layerscited by 58
- 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
- 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
- 2024Riemannian Preconditioned LoRA for Fine-Tuning Foundation Modelscited by 51
- 2023SiRA: Sparse Mixture of Low Rank Adaptationcited by 32
- 2021Federated Mixture of Expertscited by 27
- 2021Decentralized Personalized Federated Learning: Lower Bounds and Optimal Algorithm for All Personalization Modescited by 13
- 2020Learning Mixtures of Low-Rank Modelscited by 11
- 2023Provably Personalized and Robust Federated Learningcited by 11
- 2023Partially Personalized Federated Learning: Breaking the Curse of Data Heterogeneitycited by 8
- 2015Evolutionary Multimodal Optimization: A Short Surveycited by 5
- 2022PaDPaF: Partial Disentanglement with Partially-Federated GANscited by 3
Built on by
Citing papers per year: 2024: 1.
1 paper.