mlsanityData to 7 Oct 2026

Accelerating Heterogeneous Federated Learning with Closed-form Classifiers

arXiv 2406.01116 ↗First posted 3 Jun 2024cs.LG cs.AI cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) methods often struggle in highly statistically heterogeneous settings. Indeed, non-IID data distributions cause client drift and biased local solutions, particularly pronounced in the final classification layer, negatively impacting convergence speed and accuracy. To address this issue, we introduce Federated Recursive Ridge Regression (Fed3R). Our method fits a Ridge Regression classifier computed in closed form leveraging pre-trained features. Fed3R is immune to statistical heterogeneity and is invariant to the sampling order of the clients. Therefore, it proves particularly effective in cross-device scenarios. Furthermore, it is fast and efficient in terms of communication and computation costs, requiring up to two orders of magnitude fewer resources than the competitors. Finally, we propose to leverage the Fed3R parameters as an initialization for a softmax classifier and subsequently fine-tune the model using any FL algorithm (Fed3R with Fine-Tuning, Fed3R+FT). Our findings also indicate that maintaining a fixed classifier aids in stabilizing the training and learning more discriminative features in cross-device settings. Official website: https://fed-3r.github.io/.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 57 of its 89 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

57 of 89 references.

References · 57

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2021Convex optimizationcited by 2,479
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  8. 2020Adaptive Federated Optimizationcited by 1,036
  9. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  10. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  11. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  12. 2019Large Scale Incremental Learningcited by 727
38 more
  1. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  2. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  3. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  4. 2021Towards Personalized Federated Learningcited by 462
  5. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  6. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 309
  7. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  8. 2021FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Spacecited by 284
  9. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  10. 2018Applied Federated Learning: Improving Google Keyboard Query Suggestionscited by 202
  11. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 184
  12. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  13. 2016Practical Secure Aggregation for Federated Learning on User-Held Datacited by 178
  14. 2016Generalization Properties of Learning with Random Featurescited by 163
  15. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 158
  16. 2020Mime: Mimicking Centralized Stochastic Algorithms in Federated Learningcited by 152
  17. 2016Stochastic Variance Reduction for Nonconvex Optimizationcited by 143
  18. 2021Supervised Contrastive Replay: Revisiting the Nearest Class Mean Classifier in Online Class-Incremental Continual Learningcited by 140
  19. 2022PromptFL: Let Federated Participants Cooperatively Learn Prompts Instead of Models -- Federated Learning in Age of Foundation Modelcited by 133
  20. 2020Federated Visual Classification with Real-World Data Distributioncited by 126
  21. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  22. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  23. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  24. 2021FedCM: Federated Learning with Client-level Momentumcited by 77
  25. 2022Probing Representation Forgetting in Supervised and Unsupervised Continual Learningcited by 74
  26. 2021Fast Federated Learning in the Presence of Arbitrary Device Unavailabilitycited by 72
  27. 2013Divide and Conquer Kernel Ridge Regression: A Distributed Algorithm with Minimax Optimal Ratescited by 64
  28. 2022Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Featurescited by 63
  29. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  30. 2023On the Stability-Plasticity Dilemma of Class-Incremental Learningcited by 55
  31. 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
  32. 2022FedDrive: Generalizing Federated Learning to Semantic Segmentation in Autonomous Drivingcited by 41
  33. 2022FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformerscited by 33
  34. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  35. 2020FedADC: Accelerated Federated Learning with Drift Controlcited by 26
  36. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  37. 2013Distributed Coordinate Descent Method for Learning with Big Datacited by 23
  38. 2022AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimationcited by 21

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 4.

10 papers.

  1. 2024FedBaF: Federated Learning Aggregation Biased by a Foundation Modelcited by 3
  2. 2024Covariances for Free: Exploiting Mean Distributions for Training-free Federated Learningcited by 2
  3. 2025GC-Fed: Gradient Centralized Federated Learning with Partial Client Participationcited by 2
  4. 2024FedDUAL: A Dual-Strategy with Adaptive Loss and Dynamic Aggregation for Mitigating Data Heterogeneity in Federated Learningcited by 1
  5. 2026DeepAFL: Deep Analytic Federated Learningcited by 1
  6. 2026Training-Free Dynamic Upcycling of Expert Language Modelscited by 1
  7. 2025FedTDP: A Privacy-Preserving and Unified Framework for Trajectory Data Preparation via Federated Learningcited by 0
  8. 2025A Second-Order Perspective on Pruning at Initialization and Knowledge Transfercited by 0
  9. 2026IR3DE: A Linear Router for Large Language Modelscited by 0
  10. 2026Accurate and Resource-Efficient Federated Continual Learningcited by 0