mlsanityData to 14 Oct 2026

AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimation

arXiv 2204.13170 ↗First posted 27 Apr 2022cs.LG cs.AI cs.CVcited by 21

Not on the map. Why?

Abstract (arXiv)

In Federated Learning (FL), a number of clients or devices collaborate to train a model without sharing their data. Models are optimized locally at each client and further communicated to a central hub for aggregation. While FL is an appealing decentralized training paradigm, heterogeneity among data from different clients can cause the local optimization to drift away from the global objective. In order to estimate and therefore remove this drift, variance reduction techniques have been incorporated into FL optimization recently. However, these approaches inaccurately estimate the clients' drift and ultimately fail to remove it properly. In this work, we propose an adaptive algorithm that accurately estimates drift across clients. In comparison to previous works, our approach necessitates less storage and communication bandwidth, as well as lower compute costs. Additionally, our proposed methodology induces stability by constraining the norm of estimates for client drift, making it more practical for large scale FL. Experimental findings demonstrate that the proposed algorithm converges significantly faster and achieves higher accuracy than the baselines across various FL benchmarks.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 27 Apr 2022.

Builds on

29 of 39 references.

References · 29

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  3. 2018Federated Learning with Non-IID Datacited by 1,108
  4. 2019Deep Leakage from Gradientscited by 1,021
  5. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  7. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  8. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  9. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  10. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  11. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  12. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 383
17 more
  1. 2017SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradientcited by 220
  2. 2019On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimizationcited by 180
  3. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  4. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 132
  5. 2019SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentumcited by 121
  6. 2020FedSplit: An algorithmic framework for fast federated optimizationcited by 108
  7. 2021Robbing the Fed: Directly Obtaining Private Data in Federated Learning with Modified Modelscited by 108
  8. 2020On the Convergence of SGD with Biased Gradientscited by 83
  9. 2019Variance Reduced Local SGD with Lower Communication Complexitycited by 80
  10. 2014A Proximal Stochastic Gradient Method with Progressive Variance Reductioncited by 78
  11. 2020FedDANE: A Federated Newton-Type Methodcited by 76
  12. 2016AIDE: Fast and Communication Efficient Distributed Optimizationcited by 43
  13. 2021Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learningcited by 31
  14. 2015Stop Wasting My Gradients: Practical SVRGcited by 14
  15. 2015Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Settingcited by 10
  16. 2021A Variance Controlled Stochastic Method with Biased Estimation for Faster Non-convex Optimizationcited by 2
  17. 2014mS2GD: Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Settingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 4; 2024: 7; 2025: 3; 2026: 4.

21 papers.

Sort:

By area:

  1. 2022Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learningcited by 11
  2. 2024Federated Optimization with Doubly Regularized Drift Correctioncited by 10
  3. 2024Accelerating Heterogeneous Federated Learning with Closed-form Classifierscited by 10
  4. 2022On the effectiveness of partial variance reduction in federated learning with heterogeneous datacited by 9
  5. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 9
  6. 2024Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learningcited by 8
  7. 2023Window-based Model Averaging Improves Generalization in Heterogeneous Federated Learningcited by 7
  8. 2024Stabilized Proximal-Point Methods for Federated Optimizationcited by 7
  9. 2023FilFL: Client Filtering for Optimized Client Participation in Federated Learningcited by 5
  10. 2023Communication-Efficient Heterogeneous Federated Learning with Generalized Heavy-Ball Momentumcited by 3
  11. 2025GC-Fed: Gradient Centralized Federated Learning with Partial Client Participationcited by 2
  12. 2024FedSat: A Statistical Aggregation Approach for Class Imbalanced Clients in Federated Learningcited by 1
9 more
  1. 2026Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancingcited by 1
  2. 2022Mitigating Data Absence in Federated Learning Using Privacy-Controllable Data Digestscited by 0
  3. 2023DRAG: Divergence-based Adaptive Aggregation in Federated learning on Non-IID Datacited by 0
  4. 2024The Key of Parameter Skew in Federated Learningcited by 0
  5. 2025Generalizable Federated Learning using Client Adaptive Focal Modulationcited by 0
  6. 2025FedOAED: Federated On-Device Autoencoder Denoiser for Heterogeneous Data under Limited Client Availabilitycited by 0
  7. 2026FedAdaVR: Adaptive Variance Reduction for Robust Federated Learning under Limited Client Participationcited by 0
  8. 2026FedZMG: Efficient Client-Side Optimization in Federated Learningcited by 0
  9. 2026Federated Learning for Feature Generalization with Convex Constraintscited by 0