mlsanityData to 7 Oct 2026

On the effectiveness of partial variance reduction in federated learning with heterogeneous data

arXiv 2212.02191 ↗First posted 5 Dec 2022cs.LG cs.DCcited by 8

Not on the map. Why?

Abstract (arXiv)

Data heterogeneity across clients is a key challenge in federated learning. Prior works address this by either aligning client and server models or using control variates to correct client model drift. Although these methods achieve fast convergence in convex or simple non-convex problems, the performance in over-parameterized models such as deep neural networks is lacking. In this paper, we first revisit the widely used FedAvg algorithm in a deep neural network to understand how data heterogeneity influences the gradient updates across the neural network layers. We observe that while the feature extraction layers are learned efficiently by FedAvg, the substantial diversity of the final classification layers across clients impedes the performance. Motivated by this, we propose to correct model drift by variance reduction only on the final layers. We demonstrate that this significantly outperforms existing benchmarks at a similar or lower communication cost. We furthermore provide proof for the convergence rate of our algorithm.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 5 Dec 2022.

Builds on

34 of 46 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2013Intriguing properties of neural networkscited by 5,114
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2019Advances and Open Problems in Federated Learningcited by 3,021
  8. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  9. 2021Model-Contrastive Federated Learningcited by 762
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  11. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  12. 2021Federated Learning Based on Dynamic Regularizationcited by 555
22 more
  1. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 401
  2. 2020Classification with Valid and Adaptive Coveragecited by 390
  3. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 365
  4. 2020A Unified Theory of Decentralized SGD with Changing Topology and Local Updatescited by 361
  5. 2021A Field Guide to Federated Optimizationcited by 312
  6. 2020Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depthcited by 241
  7. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  8. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  9. 2021FedBABU: Towards Enhanced Representation for Federated Image Classificationcited by 180
  10. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
  11. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 131
  12. 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
  13. 2019A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descentcited by 84
  14. 2019Unified Optimal Analysis of the (Stochastic) Gradient Methodcited by 81
  15. 2022FedAvg with Fine Tuning: Local Updates Lead to Representation Learningcited by 55
  16. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  17. 2021RelaySum for Decentralized Deep Learning on Heterogeneous Datacited by 39
  18. 2019On the Discrepancy between the Theoretical Analysis and Practical Implementations of Compressed Communication for Distributed Deep Learningcited by 31
  19. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  20. 2022AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimationcited by 21
  21. 2021FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learningcited by 19
  22. 2019The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communicationcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 1; 2025: 1; 2026: 1.

8 papers.

  1. 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
  2. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  3. 2025Geometric Knowledge-Guided Localized Global Distribution Alignment for Federated Learningcited by 11
  4. 2023Scaff-PD: Communication Efficient Fair and Robust Federated Learningcited by 6
  5. 2023FedBug: A Bottom-Up Gradual Unfreezing Framework for Federated Learningcited by 3
  6. 2023Synthetic data shuffling accelerates the convergence of federated learning under data heterogeneitycited by 2
  7. 2024Federated Learning for Blind Image Super-Resolutioncited by 2
  8. 2026FedSPC: Shared Parameter Correction for Personalized Federated Learningcited by 0