mlsanityData to 7 Oct 2026

FedUV: Uniformity and Variance for Heterogeneous Federated Learning

arXiv 2402.18372 ↗First posted 27 Feb 2024cs.LG cs.AI cs.DCcited by 8

Not on the map. Why?

Abstract (arXiv)

Federated learning is a promising framework to train neural networks with widely distributed data. However, performance degrades heavily with heterogeneously distributed data. Recent work has shown this is due to the final layer of the network being most prone to local bias, some finding success freezing the final layer as an orthogonal classifier. We investigate the training dynamics of the classifier by applying SVD to the weights motivated by the observation that freezing weights results in constant singular values. We find that there are differences when training in IID and non-IID settings. Based on this finding, we introduce two regularization terms for local training to continuously emulate IID settings: (1) variance in the dimension-wise probability distribution of the classifier and (2) hyperspherical uniformity of representations of the encoder. These regularizations promote local models to act as if it were in an IID setting regardless of the local data distribution, thus offsetting proneness to bias while being flexible to the data. On extensive experiments in both label-shift and feature-shift settings, we verify that our method achieves highest performance by a large margin especially in highly non-IID cases in addition to being scalable to larger models and datasets.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

39 of 53 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  8. 2017Deep Hashing Network for Unsupervised Domain Adaptationcited by 1,181
  9. 2017Deeper, Broader and Artier Domain Generalizationcited by 983
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  11. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  12. 2021Model-Contrastive Federated Learningcited by 762
27 more
  1. 2017SphereFace: Deep Hypersphere Embedding for Face Recognitioncited by 733
  2. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  3. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  4. 2020Federated Learning with Matched Averagingcited by 564
  5. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  6. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  7. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
  8. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  9. 2021Personalized Federated Learning using Hypernetworkscited by 210
  10. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  11. 2018Hyperspherical Variational Auto-Encoderscited by 188
  12. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  13. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  14. 2020Byzantine-Robust Learning on Heterogeneous Datasets via Bucketingcited by 125
  15. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  16. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  17. 2022Layer-wised Model Aggregation for Personalized Federated Learningcited by 95
  18. 2018Learning towards Minimum Hyperspherical Energycited by 75
  19. 2018Decoupled Networkscited by 45
  20. 2021FedCor: Correlation-Based Active Client Selection Strategy for Heterogeneous Federated Learningcited by 43
  21. 2022Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentationcited by 41
  22. 2021Learning with Hyperspherical Uniformitycited by 35
  23. 2022Differentially Private Federated Learning with Local Regularization and Sparsificationcited by 35
  24. 2022CD2-pFed: Cyclic Distillation-guided Channel Decoupling for Model Personalization in Federated Learningcited by 34
  25. 2022Federated Learning with Position-Aware Neuronscited by 24
  26. 2020GradAug: A New Regularization Method for Deep Neural Networkscited by 18
  27. 2019Neural Similarity Learningcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 3.

8 papers.

  1. 2024The Impact of Cut Layer Selection in Split Federated Learningcited by 6
  2. 2025You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Datacited by 5
  3. 2024Class-Wise Federated Averaging for Efficient Personalizationcited by 0
  4. 2025Heterogeneous Federated Learning with Prototype Alignment and Upscalingcited by 0
  5. 2025A Closer Look at Personalized Fine-Tuning in Heterogeneous Federated Learningcited by 0
  6. 2026Decoupled Split Learning via Auxiliary Losscited by 0
  7. 2026Class-wise Contribution Estimation via Logit Maximization for Federated Learningcited by 0
  8. 2026From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learningcited by 0