# Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup

arXiv: https://arxiv.org/abs/2308.00522 · HTML: https://mlsanity.com/paper/2308.00522/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 61 references that are in the map.
- First posted on arXiv 30 Jul 2023.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Adaptive Federated Optimization](https://mlsanity.com/paper/2003.00295/) (2020)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification](https://mlsanity.com/paper/1909.06335/) (2019)
- [On the Convergence of Adam and Beyond](https://mlsanity.com/paper/1904.09237/) (2019)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [Federated Learning of a Mixture of Global and Local Models](https://mlsanity.com/paper/2002.05516/) (2020)
- [Don't Use Large Mini-Batches, Use Local SGD](https://mlsanity.com/paper/1808.07217/) (2018)
- [Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning](https://mlsanity.com/paper/2101.11203/) (2021)
- [On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization](https://mlsanity.com/paper/1905.03817/) (2019)
- [On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes](https://mlsanity.com/paper/1805.08114/) (2018)
- [Is Local SGD Better than Minibatch SGD?](https://mlsanity.com/paper/2002.07839/) (2020)
- [FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data](https://mlsanity.com/paper/2005.11418/) (2020)
- [A Sufficient Condition for Convergences of Adam and RMSProp](https://mlsanity.com/paper/1811.09358/) (2018)
- [SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum](https://mlsanity.com/paper/1910.00643/) (2019)
- [FedSplit: An algorithmic framework for fast federated optimization](https://mlsanity.com/paper/2005.05238/) (2020)
- [On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization](https://mlsanity.com/paper/1808.05671/) (2018)
- [First Analysis of Local GD on Heterogeneous Data](https://mlsanity.com/paper/1909.04715/) (2019)

## Built on by (most cited first, 23 of 23)

- [Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization](https://mlsanity.com/paper/2405.18890/) (2024)
- [FedMuon: Accelerating Federated Learning with Matrix Orthogonalization](https://mlsanity.com/paper/2510.27403/) (2025)
- [FedAdamW: A Communication-Efficient Optimizer with Convergence and Generalization Guarantees for Federated Large Models](https://mlsanity.com/paper/2510.27486/) (2025)
- [FADAS: Towards Federated Adaptive Asynchronous Optimization](https://mlsanity.com/paper/2407.18365/) (2024)
- [Federated Learning with Manifold Regularization and Normalized Update Reaggregation](https://mlsanity.com/paper/2311.05924/) (2023)
- [Efficient Distributed Optimization under Heavy-Tailed Noise](https://mlsanity.com/paper/2502.04164/) (2025)
- [Personalized Quantum Federated Learning for Privacy Image Classification](https://mlsanity.com/paper/2410.02547/) (2024)
- [Towards Understanding Generalization and Stability Gaps between Centralized and Decentralized Federated Learning](https://mlsanity.com/paper/2310.03461/) (2023)
- [A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needs](https://mlsanity.com/paper/2409.18915/) (2024)
- [Gradient Correction in Federated Learning with Adaptive Optimization](https://mlsanity.com/paper/2502.02727/) (2025)
- [Rethinking LoRA for Data Heterogeneous Federated Learning: Subspace and State Alignment](https://mlsanity.com/paper/2602.01746/) (2026)
- [FedSat: A Statistical Aggregation Approach for Class Imbalanced Clients in Federated Learning](https://mlsanity.com/paper/2407.03862/) (2024)
- [FedDUAL: A Dual-Strategy with Adaptive Loss and Dynamic Aggregation for Mitigating Data Heterogeneity in Federated Learning](https://mlsanity.com/paper/2412.04416/) (2024)
- [Accelerating Energy-Efficient Federated Learning in Cell-Free Networks with Adaptive Quantization](https://mlsanity.com/paper/2412.20785/) (2024)
- [DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models](https://mlsanity.com/paper/2602.19945/) (2026)
- [Convergent Differential Privacy Analysis for General Federated Learning](https://mlsanity.com/paper/2408.15621/) (2024)
- [Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum](https://mlsanity.com/paper/2504.12742/) (2025)
- [Analysis of Asynchronous Federated Learning: Unraveling the Interactions between Gradient Compression, Delay, and Data Heterogeneity](https://mlsanity.com/paper/2504.19903/) (2025)
- [LoDAdaC: a unified local training-based decentralized framework with adaptive gradients and compressed communication](https://mlsanity.com/paper/2604.09970/) (2026)
- [Rethinking the Personalized Relaxed Initialization in the Federated Learning: Consistency and Generalization](https://mlsanity.com/paper/2604.12768/) (2026)
- [Federated learning with heavy-tailed gradient noise and communication noise: a variance-reduction based algorithm](https://mlsanity.com/paper/2606.22466/) (2026)
- [FedACT: Federated Adaptive Coordinate Trust Modulation for Robust Transformer Training under Data Heterogeneity](https://mlsanity.com/paper/2607.03763/) (2026)
- [FedLNS: Leverage LayerNorm Signature Modeling to Mitigate Adversarial Manipulation in Federated LLMs](https://mlsanity.com/paper/2608.18736/) (2026)

## Abstract (arXiv)

Adaptive optimization has achieved notable success for distributed learning while extending adaptive optimizer to federated Learning (FL) suffers from severe inefficiency, including (i) rugged convergence due to inaccurate gradient estimation in global adaptive optimizer; (ii) client drifts exacerbated by local over-fitting with the local adaptive optimizer. In this work, we propose a novel momentum-based algorithm via utilizing the global gradient descent and locally adaptive amended optimizer to tackle these difficulties. Specifically, we incorporate a locally amended technique to the adaptive optimizer, named Federated Local ADaptive Amended optimizer (textitFedLADA), which estimates the global average offset in the previous communication round and corrects the local offset through a momentum-like term to further improve the empirical training speed and mitigate the heterogeneous over-fitting. Theoretically, we establish the convergence rate of textitFedLADA with a linear speedup property on the non-convex case under the partial participation settings. Moreover, we conduct extensive experiments on the real-world dataset to demonstrate the efficacy of our proposed textitFedLADA, which could greatly reduce the communication rounds and achieves higher accuracy than several baselines.