mlsanityData to 7 Oct 2026

FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learning

arXiv 2406.02355 ↗First posted 4 Jun 2024cs.CV cs.AI cs.DCcited by 2

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) has emerged as a pivotal framework for the development of effective global models (global FL) or personalized models (personalized FL) across clients with heterogeneous, non-iid data distribution. A key challenge in FL is client drift, where data heterogeneity impedes the aggregation of scattered knowledge. Recent studies have tackled the client drift issue by identifying significant divergence in the last classifier layer. To mitigate this divergence, strategies such as freezing the classifier weights and aligning the feature extractor accordingly have proven effective. Although the local alignment between classifier and feature extractor has been studied as a crucial factor in FL, we observe that it may lead the model to overemphasize the observed classes within each client. Thus, our objectives are twofold: (1) enhancing local alignment while (2) preserving the representation of unseen class samples. This approach aims to effectively integrate knowledge from individual clients, thereby improving performance for both global and personalized FL. To achieve this, we introduce a novel algorithm named FedDr+, which empowers local model alignment using dot-regression loss. FedDr+ freezes the classifier as a simplex ETF to align the features and improves aggregated global models by employing a feature distillation mechanism to retain information about unseen/missing classes. Consequently, we provide empirical evidence demonstrating that our algorithm surpasses existing methods that use a frozen classifier to boost alignment across the diverse distribution.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 53 of its 69 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

53 of 69 references.

References · 53

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  9. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  10. 2021Model-Contrastive Federated Learningcited by 762
  11. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  12. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
38 more
  1. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  2. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  3. 2020Federated Learning with Matched Averagingcited by 564
  4. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  5. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  6. 2019Federated Learning with Personalization Layerscited by 533
  7. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  8. 2022Decoupled Knowledge Distillationcited by 380
  9. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  10. 2019A Comprehensive Overhaul of Feature Distillationcited by 341
  11. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 298
  12. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  13. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  14. 2019Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivationcited by 146
  15. 2021Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillationcited by 131
  16. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  17. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  18. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  19. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  20. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  21. 2022Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learningcited by 81
  22. 2022On the Importance and Applicability of Pre-Training for Federated Learningcited by 71
  23. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  24. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 68
  25. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  26. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  27. 2017OL'E: Orthogonal Low-rank Embedding, A Plug and Play Geometric Loss for Deep Learningcited by 37
  28. 2022Switchable Online Knowledge Distillationcited by 29
  29. 2023DFRD: Data-Free Robustness Distillation for Heterogeneous Federated Learningcited by 23
  30. 2021Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networkscited by 20
  31. 2022SphereFed: Hyperspherical Federated Learningcited by 19
  32. 2018Knowledge Distillation with Feature Maps for Image Classificationcited by 17
  33. 2023FedSOL: Stabilized Orthogonal Learning with Proximal Restrictions in Federated Learningcited by 14
  34. 2024Federated Learning with Bilateral Curation for Partially Class-Disjoint Datacited by 13
  35. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  36. 2023Federated Learning with Manifold Regularization and Normalized Update Reaggregationcited by 8
  37. 2024Spectral Co-Distillation for Personalized Federated Learningcited by 8
  38. 2022It's All in the Head: Representation Knowledge Distillation through Classifier Sharingcited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2024Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learningcited by 2
  2. 2025FedRE: A Representation Entanglement Framework for Model-Heterogeneous Federated Learningcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Global Federated Learning Strategies for Building Efficient Personalized Models
  2. 2025OvA-LP: A Simple and Efficient Framework for Federated Learning on Non-IID Data
  3. 2025FedAWA: Adaptive Optimization of Aggregation Weights in Federated Learning Using Client Vectors
  4. 2024Recurrent Early Exits for Federated Learning with Heterogeneous Clients
  5. 2020Heterogeneous Federated Learning
  6. 2025FedMerge: Federated Personalization via Model Merging
  7. 2024FedTSA: A Cluster-based Two-Stage Aggregation Method for Model-heterogeneous Federated Learning
  8. 2025FedLLM-Align: Feature Extraction From Heterogeneous Clients