mlsanityData to 7 Oct 2026

FADAS: Towards Federated Adaptive Asynchronous Optimization

arXiv 2407.18365 ↗First posted 25 Jul 2024cs.LG cs.AI cs.DCcited by 10

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) has emerged as a widely adopted training paradigm for privacy-preserving machine learning. While the SGD-based FL algorithms have demonstrated considerable success in the past, there is a growing trend towards adopting adaptive federated optimization methods, particularly for training large-scale models. However, the conventional synchronous aggregation design poses a significant challenge to the practical deployment of those adaptive federated optimization methods, particularly in the presence of straggler clients. To fill this research gap, this paper introduces federated adaptive asynchronous optimization, named FADAS, a novel method that incorporates asynchronous updates into adaptive federated optimization with provable guarantees. To further enhance the efficiency and resilience of our proposed method in scenarios with significant asynchronous delays, we also extend FADAS with a delay-adaptive learning adjustment strategy. We rigorously establish the convergence rate of the proposed algorithms and empirical results demonstrate the superior performance of FADAS over other asynchronous FL baselines.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

40 of 55 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2020Adaptive Federated Optimizationcited by 1,036
  7. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  8. 2019On the Convergence of Adam and Beyondcited by 834
  9. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  10. 2020Federated Learning with Matched Averagingcited by 564
  11. 2021Federated Learning Based on Dynamic Regularizationcited by 555
  12. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 349
28 more
  1. 2019Asynchronous Federated Optimizationcited by 264
  2. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  3. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
  4. 2021Federated Learning with Buffered Asynchronous Aggregationcited by 188
  5. 2018On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimizationcited by 173
  6. 2018SGD and Hogwild! Convergence Without the Bounded Gradients Assumptioncited by 94
  7. 2018Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networkscited by 89
  8. 2020VAFL: a Method of Vertical Asynchronous Federated Learningcited by 87
  9. 2022Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learningcited by 76
  10. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 68
  11. 2015Perturbed Iterate Analysis for Asynchronous Stochastic Optimizationcited by 67
  12. 2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
  13. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 60
  14. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  15. 2021Anarchic Federated Learningcited by 42
  16. 2022Faster Adaptive Federated Learningcited by 35
  17. 2018Improved asynchronous parallel optimization analysis for stochastic incremental methodscited by 30
  18. 2022Accelerated Federated Learning with Decoupled Adaptive Optimizationcited by 23
  19. 2023Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedupcited by 23
  20. 2022SWIFT: Rapid Decentralized Federated Learning via Wait-Free Model Communicationcited by 21
  21. 2021Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updatescited by 16
  22. 2020Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Datacited by 15
  23. 2021Communication-Compressed Adaptive Gradient Method for Distributed Nonconvex Optimizationcited by 12
  24. 2022Delay-adaptive step-sizes for asynchronous learningcited by 11
  25. 2022Unbounded Gradients in Federated Learning with Buffered Asynchronous Aggregationcited by 11
  26. 2023Asynchronous Federated Learning with Bidirectional Quantized Communications and Buffered Aggregationcited by 10
  27. 2023FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averagingcited by 2
  28. 2023FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Datacited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 3.

10 papers.

  1. 2024Understanding the Statistical Accuracy-Communication Trade-off in Personalized Federated Learning with Minimax Guaranteescited by 1
  2. 2025Guided Model Merging for Hybrid Data Learning: Leveraging Centralized Data to Refine Decentralized Modelscited by 0
  3. 2025Analysis of Asynchronous Federated Learning: Unraveling the Interactions between Gradient Compression, Delay, and Data Heterogeneitycited by 0
  4. 2025Hybrid Batch Normalisation: Resolving the Dilemma of Batch Normalisation in Federated Learningcited by 0
  5. 2025Why Federated Optimization Fails to Achieve Perfect Fitting? A Theoretical Perspective on Client-Side Optimacited by 0
  6. 2025ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Modelscited by 0
  7. 2025Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learningcited by 0
  8. 2026Ordered Local Momentum for Asynchronous Distributed Learning under Arbitrary Delayscited by 0
  9. 2026AlignFed: Alignment-Aware Asynchronous Federated Fine-Tuning for Large Language Models in Heterogeneous Edge Environmentscited by 0
  10. 2026AutoLoCo: Communication Efficient Distributed LLM Training via Adaptive Synchronizationcited by 0