mlsanityData to 14 Oct 2026

Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning

arXiv 2210.00226 ↗First posted 1 Oct 2022cs.LGcited by 46

Not on the map. Why?

Abstract (arXiv)

Federated learning aims to train models collaboratively across different clients without the sharing of data for privacy considerations. However, one major challenge for this learning paradigm is the em data heterogeneity problem, which refers to the discrepancies between the local data distributions among various clients. To tackle this problem, we first study how data heterogeneity affects the representations of the globally aggregated models. Interestingly, we find that heterogeneous data results in the global model suffering from severe em dimensional collapse, in which representations tend to reside in a lower-dimensional space instead of the ambient space. Moreover, we observe a similar phenomenon on models locally trained on each client and deduce that the dimensional collapse on the global model is inherited from local models. In addition, we theoretically analyze the gradient flow dynamics to shed light on how data heterogeneity result in dimensional collapse for local models. To remedy this problem caused by the data heterogeneity, we propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning. Specifically, sc FedDecorr applies a regularization term during local training that encourages different dimensions of representations to be uncorrelated. sc FedDecorr, which is implementation-friendly and computationally-efficient, yields consistent improvements over baselines on standard benchmark datasets. Code: https://github.com/bytedance/FedDecorr.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Builds on 32 of its 37 references that are in the map. First posted on arXiv 1 Oct 2022.

Builds on

32 of 37 references.

References · 32

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  5. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  8. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  9. 2021Model-Contrastive Federated Learningcited by 766
  10. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  11. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  12. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
20 more
  1. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  2. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  3. 2020Federated Learning with Matched Averagingcited by 564
  4. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  5. 2019Federated Learning with Personalization Layerscited by 541
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  7. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  8. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
  9. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  10. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 283
  11. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  12. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  13. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  14. 2021On Feature Decorrelation in Self-Supervised Learningcited by 160
  15. 2020Lower Bounds and Optimal Algorithms for Personalized Federated Learningcited by 123
  16. 2020Revisiting Training Strategies and Generalization Performance in Deep Metric Learningcited by 100
  17. 2015Reducing Overfitting in Deep Networks by Decorrelating Representationscited by 94
  18. 2018Decorrelated Batch Normalizationcited by 83
  19. 2020Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithmscited by 78
  20. 2021Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learningcited by 49

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 13; 2024: 13; 2025: 16; 2026: 4.

46 papers.

Sort:

By area:

  1. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  2. 2023Diffusion Model as Representation Learnercited by 95
  3. 2023Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Datacited by 35
  4. 2024Relaxed Contrastive Learning for Federated Learningcited by 32
  5. 2024Fine-Tuning is Fine, if Calibratedcited by 24
  6. 2023FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clusteringcited by 15
  7. 2024Rethinking the Representation in Federated Unsupervised Learning with Non-IID Datacited by 13
  8. 2025Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learningcited by 12
  9. 2024Distributed Federated Learning-Based Deep Learning Model for Privacy MRI Brain Tumor Detectioncited by 9
  10. 2023FedFN: Feature Normalization for Alleviating Data Heterogeneity Problem in Federated Learningcited by 8
  11. 2025Quantum Federated Learning for Multimodal Data: A Modality-Agnostic Approachcited by 8
  12. 2023Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Datacited by 7
34 more
  1. 2023Fake It Till Make It: Federated Learning with Consensus-Oriented Generationcited by 6
  2. 2025Federated Learning for Medical Image Classification: A Comprehensive Benchmarkcited by 6
  3. 2023Tackling the Non-IID Issue in Heterogeneous Federated Learning by Gradient Harmonizationcited by 4
  4. 2024Towards Eliminating Hard Label Constraints in Gradient Inversion Attackscited by 4
  5. 2024Enhancing Data Quality in Federated Fine-Tuning of Foundation Modelscited by 4
  6. 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learningcited by 4
  7. 2023FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learningcited by 3
  8. 2023Federated Learning Empowered by Generative Contentcited by 3
  9. 2024SCALA: Split Federated Learning with Concatenated Activations and Logit Adjustmentscited by 3
  10. 2024Virtual Nodes Can Help: Tackling Distribution Shifts in Federated Graph Learningcited by 3
  11. 2025Personalized Federated Learning via Dual-Prompt Optimization and Cross Fusioncited by 3
  12. 2023Understanding the Role of Layer Normalization in Label-Skewed Federated Learningcited by 2
  13. 2023Information-Theoretic Generalization Analysis for Topology-aware Heterogeneous Federated Edge Learning over Noisy Channelscited by 2
  14. 2025GC-Fed: Gradient Centralized Federated Learning with Partial Client Participationcited by 2
  15. 2023FedMKGC: Privacy-Preserving Federated Multilingual Knowledge Graph Completioncited by 1
  16. 2024The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustnesscited by 1
  17. 2024Federated Cross-Training Learners for Robust Generalization under Data Heterogeneitycited by 1
  18. 2024Tackling Dimensional Collapse toward Comprehensive Universal Domain Adaptationcited by 1
  19. 2025Federated Self-Supervised Learning for One-Shot Cross-Modal and Cross-Imaging Technique Segmentationcited by 1
  20. 2025Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregationcited by 1
  21. 2025FedWSQ: Efficient Federated Learning with Weight Standardization and Distribution-Aware Non-Uniform Quantizationcited by 1
  22. 2026ForgeVLA: Federated Vision-Language-Action Learning without Language Annotationscited by 1
  23. 2024Approximate Gradient Coding for Privacy-Flexible Federated Learning with Non-IID Datacited by 0
  24. 2024Client2Vec: Improving Federated Learning by Distribution Shifts Aware Client Indexingcited by 0
  25. 2025Invariant Federated Learning for Edge Intelligence: Mitigating Heterogeneity and Asynchrony via Exit Strategy and Invariant Penaltycited by 0
  26. 2025FedGAI: Federated Style Learning with Cloud-Edge Collaboration for Generative AI in Fashion Designcited by 0
  27. 2025Enhancing the Performance of Global Model by Improving the Adaptability of Local Models in Federated Learningcited by 0
  28. 2025ATR-Bench: A Federated Learning Benchmark for Adaptation, Trust, and Reasoningcited by 0
  29. 2025Soft Separation and Distillation: Toward Global Uniformity in Federated Unsupervised Learningcited by 0
  30. 2025Calibrating Biased Distribution in VFM-derived Latent Space via Cross-Domain Geometric Consistencycited by 0
  31. 2025Cooperative Pseudo Labeling for Unsupervised Federated Classificationcited by 0
  32. 2026Adaptive Dual-Weighting Framework for Federated Learning via Out-of-Distribution Detectioncited by 0
  33. 2026Global Federated Learning Strategies for Building Efficient Personalized Modelscited by 0
  34. 2026Understanding Head Geometry and Dynamics in Federated Regression through a Natural Solution Selection Rule: An Unconstrained Feature Model Analysiscited by 0