mlsanityData to 14 Oct 2026

Federated Optimization with Doubly Regularized Drift Correction

arXiv 2404.08447 ↗First posted 12 Apr 2024cs.LG math.OCcited by 10

Not on the map. Why?

Abstract (arXiv)

Federated learning is a distributed optimization paradigm that allows training machine learning models across decentralized devices while keeping the data localized. The standard method, FedAvg, suffers from client drift which can hamper performance and increase communication costs over centralized methods. Previous works proposed various strategies to mitigate drift, yet none have shown uniformly improved communication-computation trade-offs over vanilla gradient descent. In this work, we revisit DANE, an established method in distributed optimization. We show that (i) DANE can achieve the desired communication reduction under Hessian similarity constraints. Furthermore, (ii) we present an extension, DANE+, which supports arbitrary inexact local solvers and has more freedom to choose how to aggregate the local updates. We propose (iii) a novel method, FedRed, which has improved local computational complexity and retains the same communication complexity compared to DANE/DANE+. This is achieved by using doubly regularized drift correction.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 24 of its 37 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

24 of 37 references.

References · 24

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Advances and Open Problems in Federated Learningcited by 3,046
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  6. 2014Convex Optimization: Algorithms and Complexitycited by 575
  7. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  8. 2021Federated Learning Based on Dynamic Regularizationcited by 562
  9. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  10. 2020Minibatch vs Local SGD for Heterogeneous Distributed Learningcited by 147
  11. 2020FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Datacited by 144
  12. 2022ProxSkip: Yes! Local Gradient Steps Provably Lead to Communication Acceleration! Finally!cited by 134
12 more
  1. 2013Communication Efficient Distributed Optimization using an Approximate Newton-type Methodcited by 132
  2. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 123
  3. 2019First Analysis of Local GD on Heterogeneous Datacited by 81
  4. 2018On the Ineffectiveness of Variance Reduced Optimization for Deep Learningcited by 48
  5. 2017GIANT: Globally Improved Approximate Newton Method for Distributed Optimizationcited by 44
  6. 2020Statistically Preconditioned Accelerated Gradient Method for Distributed Optimizationcited by 30
  7. 2022RandProx: Primal-Dual Optimization Algorithms with Randomized Proximal Updatescited by 28
  8. 2022Can 5th Generation Local Training Methods Support Client Sampling? Yes!cited by 22
  9. 2022AdaBest: Minimizing Client Drift in Federated Learning via Adaptive Bias Estimationcited by 21
  10. 2019On Convergence of Distributed Approximate Newton Methods: Globalization, Sharper Bounds and Beyondcited by 18
  11. 2022Faster federated optimization under second-order similaritycited by 18
  12. 2021Acceleration in Distributed Optimization under Similaritycited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 1.

10 papers.

  1. 2025FedMuon: Federated Learning with Bias-corrected LMO-based Optimizationcited by 8
  2. 2024Stabilized Proximal-Point Methods for Federated Optimizationcited by 7
  3. 2024Hierarchical Federated Learning with Multi-Timescale Gradient Correctioncited by 7
  4. 2025Revisiting LocalSGD and SCAFFOLD: Improved Rates and Missing Analysiscited by 4
  5. 2025Accelerated Distributed Optimization with Compression and Error Feedbackcited by 4
  6. 2025Exploiting Similarity for Computation and Communication-Efficient Decentralized Optimizationcited by 1
  7. 2025Non-Convex Federated Optimization under Cost-Aware Client Selectioncited by 1
  8. 2025Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optimacited by 0
  9. 2025Beyond Scaffold: A Unified Spatio-Temporal Gradient Tracking Methodcited by 0
  10. 2026Heterogeneity-Aware Client Selection Methodology For Efficient Federated Learningcited by 0