mlsanityData to 14 Oct 2026

Efficient Adaptive Federated Optimization

arXiv 2410.18117 ↗First posted 10 Oct 2024cs.LG cs.DC math.OCcited by 2

Not on the map. Why?

Abstract (arXiv)

Adaptive optimization is critical in federated learning, where enabling adaptivity on both the server and client sides has proven essential for achieving optimal performance. However, the scalability of such jointly adaptive systems is often hindered by resource limitations in communication and memory. In this paper, we introduce a class of efficient adaptive algorithms, named FedAda2 and its enhanced version FedAda2++, designed specifically for large-scale, cross-device federated environments. FedAda2 optimizes communication efficiency by avoiding the transfer of preconditioners between the server and clients. Additionally, FedAda2++ extends this approach by incorporating memory-efficient adaptive optimizers on the client side, further reducing on-device memory usage. Theoretically, we demonstrate that FedAda2 and FedAda2++ achieve the same convergence rates for general, non-convex objectives as its more resource-intensive counterparts that directly integrate joint adaptivity. Extensive empirical evaluations on image and text datasets demonstrate both the advantages of joint adaptivity and the effectiveness of FedAda2/FedAda2++.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

25 of 33 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2019Advances and Open Problems in Federated Learningcited by 3,046
  6. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  7. 2020Adaptive Federated Optimizationcited by 1,046
  8. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2019On the Convergence of Adam and Beyondcited by 841
  11. 2021ImageNet-21K Pretraining for the Massescited by 613
  12. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
13 more
  1. 2021A Field Guide to Federated Optimizationcited by 315
  2. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  3. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  4. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  5. 2019First Exit Time Analysis of Stochastic Gradient Descent Under Heavy-Tailed Gradient Noisecited by 44
  6. 2020Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noisecited by 33
  7. 2021Local Adaptivity in Federated Learning: Convergence and Consistencycited by 31
  8. 2019Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Ratescited by 26
  9. 2021Toward Communication Efficient Adaptive Gradient Methodcited by 25
  10. 2022Differentially Private Adaptive Optimization with Delayed Preconditionerscited by 18
  11. 2020Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Datacited by 15
  12. 2019Extreme Tensoring for Low-Memory Preconditioningcited by 5
  13. 2023FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Datacited by 2

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Efficient Distributed Optimization under Heavy-Tailed Noisecited by 7
  2. 2025FedDuA: Doubly Adaptive Federated Learningcited by 0