# Faster Adaptive Federated Learning

arXiv: https://arxiv.org/abs/2212.00974 · HTML: https://mlsanity.com/paper/2212.00974/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 56 references that are in the map.
- First posted on arXiv 2 Dec 2022.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Bag of Tricks for Image Classification with Convolutional Neural Networks](https://mlsanity.com/paper/1812.01187/) (2018)
- [Momentum-Based Variance Reduction in Non-Convex SGD](https://mlsanity.com/paper/1905.10018/) (2019)
- [SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator](https://mlsanity.com/paper/1807.01695/) (2018)
- [Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning](https://mlsanity.com/paper/1807.06629/) (2018)
- [Tighter Theory for Local SGD on Identical and Heterogeneous Data](https://mlsanity.com/paper/1909.04746/) (2019)
- [Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning](https://mlsanity.com/paper/2101.11203/) (2021)
- [AdaGrad stepsizes: Sharp convergence over nonconvex landscapes](https://mlsanity.com/paper/1806.01811/) (2018)
- [On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization](https://mlsanity.com/paper/1905.03817/) (2019)
- [Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning](https://mlsanity.com/paper/2008.03606/) (2020)
- [Is Local SGD Better than Minibatch SGD?](https://mlsanity.com/paper/2002.07839/) (2020)
- [Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods](https://mlsanity.com/paper/1902.08297/) (2019)
- [Communication-Efficient Adaptive Federated Learning](https://mlsanity.com/paper/2205.02719/) (2022)
- [Faster Non-Convex Federated Learning via Global and Local Momentum](https://mlsanity.com/paper/2012.04061/) (2020)
- [STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learning](https://mlsanity.com/paper/2106.10435/) (2021)
- [Closing the Generalization Gap of Cross-silo Federated Medical Image Segmentation](https://mlsanity.com/paper/2203.10144/) (2022)
- [SUPER-ADAM: Faster and Universal Framework of Adaptive Gradients](https://mlsanity.com/paper/2106.08208/) (2021)
- [Why are Adaptive Methods Good for Attention Models?](https://mlsanity.com/paper/1912.03194/) (2019)
- [Escaping Saddle Points with Adaptive Gradient Methods](https://mlsanity.com/paper/1901.09149/) (2019)
- [Fast OSCAR and OWL Regression via Safe Screening Rules](https://mlsanity.com/paper/2006.16433/) (2020)
- [Toward Communication Efficient Adaptive Gradient Method](https://mlsanity.com/paper/2109.05109/) (2021)
- [RetrievalGuard: Provably Robust 1-Nearest Neighbor Image Retrieval](https://mlsanity.com/paper/2206.11225/) (2022)
- [Coordinating Momenta for Cross-silo Federated Learning](https://mlsanity.com/paper/2102.03970/) (2021)
- [Multi-Classifier Interactive Learning for Ambiguous Speech Emotion Recognition](https://mlsanity.com/paper/2012.05429/) (2020)
- [Inflating Topic Relevance with Ideology: A Case Study of Political Ideology Bias in Social Topic Detection Models](https://mlsanity.com/paper/2011.14293/) (2020)

## Built on by (most cited first, 30 of 35)

- [AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networks](https://mlsanity.com/paper/2403.13101/) (2024)
- [Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction](https://mlsanity.com/paper/2310.08670/) (2023)
- [Adversarial Weight Perturbation Improves Generalization in Graph Neural Networks](https://mlsanity.com/paper/2212.04983/) (2022)
- [Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch](https://mlsanity.com/paper/2403.14729/) (2024)
- [Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality](https://mlsanity.com/paper/2406.11048/) (2024)
- [Solving a Class of Non-Convex Minimax Optimization in Federated Learning](https://mlsanity.com/paper/2310.03613/) (2023)
- [FADAS: Towards Federated Adaptive Asynchronous Optimization](https://mlsanity.com/paper/2407.18365/) (2024)
- [Decentralized Riemannian Algorithm for Nonconvex Minimax Problems](https://mlsanity.com/paper/2302.03825/) (2023)
- [On Provable Benefits of Muon in Federated Learning](https://mlsanity.com/paper/2510.03866/) (2025)
- [FedLion: Faster Adaptive Federated Optimization with Fewer Communication](https://mlsanity.com/paper/2402.09941/) (2024)
- [FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learning](https://mlsanity.com/paper/2405.11811/) (2024)
- [Federated Conditional Stochastic Optimization](https://mlsanity.com/paper/2310.02524/) (2023)
- [Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspective](https://mlsanity.com/paper/2405.20431/) (2024)
- [Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2408.14736/) (2024)
- [Serverless Federated AUPRC Optimization for Multi-Party Collaborative Imbalanced Data Mining](https://mlsanity.com/paper/2308.03035/) (2023)
- [On-ramp and Off-ramp Traffic Flows Estimation Based on A Data-driven Transfer Learning Framework](https://mlsanity.com/paper/2308.03538/) (2023)
- [Boosting Federated Domain Generalization: Understanding the Role of Advanced Pre-Trained Architectures](https://mlsanity.com/paper/2409.13527/) (2024)
- [Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practice](https://mlsanity.com/paper/2510.12595/) (2025)
- [Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment](https://mlsanity.com/paper/2602.01746/) (2026)
- [Performance and Energy Consumption of Parallel Machine Learning Algorithms](https://mlsanity.com/paper/2305.00798/) (2023)
- [Federated Knowledge Graph Unlearning via Diffusion Model](https://mlsanity.com/paper/2403.08554/) (2024)
- [Addressing Heterogeneity in Federated Load Forecasting with Personalization Layers](https://mlsanity.com/paper/2404.01517/) (2024)
- [Addressing Label Shift in Distributed Learning via Entropy Regularization](https://mlsanity.com/paper/2502.02544/) (2025)
- [Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data](https://mlsanity.com/paper/2508.20557/) (2025)
- [A Byzantine-Robust Federated Learning with Nesterov Momentum](https://mlsanity.com/paper/2511.02657/) (2025)
- [DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models](https://mlsanity.com/paper/2602.19945/) (2026)
- [FedSiKD: Clients Similarity and Knowledge Distillation: Addressing Non-i.i.d. and Constraints in Federated Learning](https://mlsanity.com/paper/2402.09095/) (2024)
- [A Whole-Process Certifiably Robust Aggregation Method Against Backdoor Attacks in Federated Learning](https://mlsanity.com/paper/2407.00719/) (2024)
- [Painless Federated Learning: An Interplay of Line-Search and Extrapolation](https://mlsanity.com/paper/2408.17145/) (2024)
- [FedGTST: Boosting Global Transferability of Federated Models via Statistics Tuning](https://mlsanity.com/paper/2410.13045/) (2024)

## Abstract (arXiv)

Federated learning has attracted increasing attention with the emergence of distributed data. While extensive federated learning algorithms have been proposed for the non-convex distributed problem, federated learning in practice still faces numerous challenges, such as the large training iterations to converge since the sizes of models and datasets keep increasing, and the lack of adaptivity by SGD-based model updates. Meanwhile, the study of adaptive methods in federated learning is scarce and existing works either lack a complete theoretical convergence guarantee or have slow sample complexity. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on the momentum-based variance-reduced technique in cross-silo FL. We first explore how to design the adaptive algorithm in the FL setting. By providing a counter-example, we prove that a simple combination of FL and adaptive methods could lead to divergence. More importantly, we provide a convergence analysis for our method and prove that our algorithm is the first adaptive FL algorithm to reach the best-known samples O(ε-3) and O(ε-2) communication rounds to find an ε-stationary point without large batches. The experimental results on the language modeling task and image classification task with heterogeneous data demonstrate the efficiency of our algorithms.