mlsanityData to 14 Oct 2026

FediOS: Decoupling Orthogonal Subspaces for Personalization in Feature-skew Federated Learning

arXiv 2311.18559 ↗First posted 30 Nov 2023cs.LG cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Personalized federated learning (pFL) enables collaborative training among multiple clients to enhance the capability of customized local models. In pFL, clients may have heterogeneous (also known as non-IID) data, which poses a key challenge in how to decouple the data knowledge into generic knowledge for global sharing and personalized knowledge for preserving local personalization. A typical way of pFL focuses on label distribution skew, and they adopt a decoupling scheme where the model is split into a common feature extractor and two prediction heads (generic and personalized). However, such a decoupling scheme cannot solve the essential problem of feature skew heterogeneity, because a common feature extractor cannot decouple the generic and personalized features. Therefore, in this paper, we rethink the architecture decoupling design for feature-skew pFL and propose an effective pFL method called FediOS. In FediOS, we reformulate the decoupling into two feature extractors (generic and personalized) and one shared prediction head. Orthogonal projections are used for clients to map the generic features into one common subspace and scatter the personalized features into different subspaces to achieve decoupling for them. In addition, a shared prediction head is trained to balance the importance of generic and personalized features during inference. Extensive experiments on four vision datasets demonstrate our method reaches state-of-the-art pFL performances under feature skew heterogeneity.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 48 of its 71 references that are in the map. First posted on arXiv 30 Nov 2023.

Builds on

48 of 71 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2016Densely Connected Convolutional Networkscited by 6,720
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  7. 2016Wide Residual Networkscited by 3,070
  8. 2019Advances and Open Problems in Federated Learningcited by 3,046
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  11. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,331
  12. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
36 more
  1. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  2. 2021Model-Contrastive Federated Learningcited by 766
  3. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  4. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  5. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  6. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  7. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  8. 2020An Efficient Framework for Clustered Federated Learningcited by 511
  9. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 503
  10. 2021Towards Personalized Federated Learningcited by 464
  11. 2019Orthogonal Gradient Descent for Continual Learningcited by 369
  12. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  13. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  14. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 182
  15. 2020Survey of Personalization Techniques for Federated Learningcited by 145
  16. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  17. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 109
  18. 2020Continual Learning in Low-rank Orthogonal Subspacescited by 99
  19. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  20. 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 67
  21. 2023FedDisco: Federated Learning with Discrepancy-Aware Collaborationcited by 67
  22. 2020FedHome: Cloud-Edge based Personalized Federated Learning for In-Home Health Monitoringcited by 61
  23. 2022FedCorr: Multi-Stage Federated Learning for Label Noise Correctioncited by 61
  24. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
  25. 2020Guided Variational Autoencoder for Disentanglement Learningcited by 57
  26. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  27. 2022Test-Time Robust Personalization for Federated Learningcited by 39
  28. 2023Federated Incremental Semantic Segmentationcited by 36
  29. 2023FedCP: Separating Feature Information for Personalized Federated Learning via Conditional Policycited by 36
  30. 2022Disentangled Federated Learning for Tackling Attributes Skew via Invariant Aggregation and Diversity Transferringcited by 28
  31. 2021Personalized Retrogress-Resilient Framework for Real-World Medical Federated Learningcited by 25
  32. 2022Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matchingcited by 24
  33. 2023Confidence-aware Personalized Federated Learning via Variational Expectation Maximizationcited by 17
  34. 2021Unified Group Fairness on Federated Learningcited by 15
  35. 2023Learning Cautiously in Federated Learning with Noisy and Heterogeneous Clientscited by 12
  36. 2014Subspace Alignment For Domain Adaptationcited by 10

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3; 2025: 1.

5 papers.

  1. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 13
  2. 2024Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalizationcited by 10
  3. 2025FedORGP: Guiding Heterogeneous Federated Learning with Orthogonality Regularization on Global Prototypescited by 1
  4. 2023Feature Norm Regularized Federated Learning: Transforming Skewed Distributions into Global Insightscited by 0
  5. 2024FedGuCci: Making Local Models More Connected in Landscape for Federated Learningcited by 0