Factor-Assisted Federated Learning for Personalized Optimization with Heterogeneous Data
Not on the map. Why?
Abstract (arXiv)
Federated learning is an emerging distributed machine learning framework aiming at protecting data privacy. Data heterogeneity is one of the core challenges in federated learning, which could severely degrade the convergence rate and prediction performance of deep neural networks. To address this issue, we develop a novel personalized federated learning framework for heterogeneous data, which we refer to as FedSplit. This modeling framework is motivated by the finding that, data in different clients contain both common knowledge and personalized knowledge. Then the hidden elements in each neural layer can be split into the shared and personalized groups. With this decomposition, a novel objective function is established and optimized. We demonstrate FedSplit enjoyers a faster convergence speed than the standard federated learning method both theoretically and empirically. The generalization bound of the FedSplit method is also studied. To practically implement the proposed method on real datasets, factor analysis is introduced to facilitate the decoupling of hidden elements. This leads to a practically implemented model for FedSplit and we further refer to as FedFac. We demonstrated by simulation studies that, using factor analysis can well recover the underlying shared/personalized decomposition. The superior prediction performance of FedFac is further verified empirically by comparison with various state-of-the-art federated learning methods on several real datasets.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 51 of its 70 references that are in the map. First posted on arXiv 7 Dec 2023.
Builds on
51 of 70 references.
References · 51
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
- 2019Advances and Open Problems in Federated Learningcited by 3,046
- 2016Deep Learning with Differential Privacycited by 2,728
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
- 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
- 2020Adaptive Federated Optimizationcited by 1,046
- 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
- 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
- 2018LEAF: A Benchmark for Federated Settingscited by 792
- 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
- 2017Federated Multi-Task Learningcited by 672
38 more
- 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
- 2020Personalized Federated Learning with Moreau Envelopescited by 618
- 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 574
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
- 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
- 2021Federated Learning Based on Dynamic Regularizationcited by 562
- 2019Federated Learning with Personalization Layerscited by 541
- 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
- 2020An Efficient Framework for Clustered Federated Learningcited by 511
- 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
- 2021Towards Personalized Federated Learningcited by 464
- 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
- 2019Agnostic Federated Learningcited by 456
- 2019Fair Resource Allocation in Federated Learningcited by 394
- 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
- 2020Adaptive Personalized Federated Learningcited by 358
- 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 325
- 2020Three Approaches for Personalization with Applications to Federated Learningcited by 325
- 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 283
- 2019Improving Federated Learning Personalization via Model Agnostic Meta Learningcited by 269
- 2020Federated Learning of a Mixture of Global and Local Modelscited by 210
- 2021Federated Multi-Task Learning under a Mixture of Distributionscited by 192
- 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
- 2019Federated Evaluation of On-device Personalizationcited by 147
- 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
- 2020Salvaging Federated Learning by Local Adaptationcited by 142
- 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
- 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 122
- 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
- 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 96
- 2021Differentially Private Federated Learning on Heterogeneous Datacited by 74
- 2019Variational Federated Multi-Task Learningcited by 69
- 2021Personalized Federated Learning through Local Memorizationcited by 57
- 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 56
- 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
- 2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
- 2020FedNER: Privacy-preserving Medical Named Entity Recognition with Federated Learningcited by 24
- 2021Neural Tangent Kernel Empowered Federated Learningcited by 15
50 most cited of 51 All 51 (JSON)
Built on by
Not cited yet.