mlsanityData to 7 Oct 2026

Decoupling General and Personalized Knowledge in Federated Learning via Additive and Low-Rank Decomposition

arXiv 2406.19931 ↗First posted 28 Jun 2024cs.LG cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

To address data heterogeneity, the key strategy of Personalized Federated Learning (PFL) is to decouple general knowledge (shared among clients) and client-specific knowledge, as the latter can have a negative impact on collaboration if not removed. Existing PFL methods primarily adopt a parameter partitioning approach, where the parameters of a model are designated as one of two types: parameters shared with other clients to extract general knowledge and parameters retained locally to learn client-specific knowledge. However, as these two types of parameters are put together like a jigsaw puzzle into a single model during the training process, each parameter may simultaneously absorb both general and client-specific knowledge, thus struggling to separate the two types of knowledge effectively. In this paper, we introduce FedDecomp, a simple but effective PFL paradigm that employs parameter additive decomposition to address this issue. Instead of assigning each parameter of a model as either a shared or personalized one, FedDecomp decomposes each parameter into the sum of two parameters: a shared one and a personalized one, thus achieving a more thorough decoupling of shared and personalized knowledge compared to the parameter partitioning method. In addition, as we find that retaining local knowledge of specific clients requires much lower model capacity compared with general knowledge across all clients, we let the matrix containing personalized parameters be low rank during the training process. Moreover, a new alternating training strategy is proposed to further improve the performance. Experimental results across multiple datasets and varying degrees of data heterogeneity demonstrate that FedDecomp outperforms state-of-the-art methods up to 4.9%. The code is available at https://github.com/XinghaoWu/FedDecomp.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 25 of its 50 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

25 of 50 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019Deep Leakage from Gradientscited by 1,004
  3. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  4. 2021Model-Contrastive Federated Learningcited by 762
  5. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 707
  6. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  7. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  8. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  9. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  10. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  11. 2019Federated Learning with Personalization Layerscited by 533
  12. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
13 more
  1. 2020Think Locally, Act Globally: Federated Learning with Local and Global Representationscited by 322
  2. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  3. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  4. 2022Federated Learning with Partial Model Personalizationcited by 114
  5. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  6. 2020Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computingcited by 52
  7. 2023Efficient Personalized Federated Learning via Sparse Model-Adaptationcited by 38
  8. 2023Bold but Cautious: Unlocking the Potential of Personalized Federated Learning through Cautiously Aggressive Collaborationcited by 23
  9. 2024Multi-level Personalized Federated Learning on Heterogeneous and Long-Tailed Datacited by 14
  10. 2023Learning Cautiously in Federated Learning with Noisy and Heterogeneous Clientscited by 12
  11. 2024Tackling Noisy Labels with Network Parameter Additive Decompositioncited by 8
  12. 2024Estimating before Debiasing: A Bayesian Approach to Detaching Prior Bias in Federated Semi-Supervised Learningcited by 5
  13. 2023Take Your Pick: Enabling Effective Personalized Federated Learning within Low-dimensional Feature Spacecited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 3.

15 papers.

Sort:

By area:

  1. 2024Why Go Full? Elevating Federated Learning Through Partial Network Updatescited by 6
  2. 2025The Panaceas for Improving Low-Rank Decomposition in Communication-Efficient Federated Learningcited by 5
  3. 2025Beyond Higher Rank: Token-wise Input-Output Projections for Efficient Low-Rank Adaptationcited by 5
  4. 2024Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformationcited by 2
  5. 2024DualFed: Enjoying both Generalization and Personalization in Federated Learning via Hierachical Representationscited by 2
  6. 2025Enhancing Visual Representation with Textual Semantics: Textual Semantics-Powered Prototypes for Heterogeneous Federated Learningcited by 2
  7. 2025Choice Outweighs Effort: Facilitating Complementary Knowledge Fusion in Federated Learning via Re-calibration and Merit-discriminationcited by 1
  8. 2025FedBiF: Communication-Efficient Federated Learning via Bits Freezingcited by 1
  9. 2025Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense Evaluationcited by 1
  10. 2025ML Mule: Mobile-Driven Context-Aware Collaborative Learningcited by 0
  11. 2025TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environmentscited by 0
  12. 2025Prototype-Guided and Lightweight Adapters for Inherent Interpretation and Generalisation in Federated Learningcited by 0
3 more
  1. 2026PiXTime: A Model for Federated Time Series Forecasting with Heterogeneous Data across Nodescited by 0
  2. 2026Fed-GAME: Personalized Federated Learning with Graph Attention Mixture-of-Experts For Time-Series Forecastingcited by 0
  3. 2026From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learningcited by 0