# Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization

arXiv: https://arxiv.org/abs/2405.17932 · HTML: https://mlsanity.com/paper/2405.17932/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 63 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 30 of 31)

- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Bayesian Nonparametric Federated Learning of Neural Networks](https://mlsanity.com/paper/1905.12022/) (2019)
- [Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach](https://mlsanity.com/paper/2001.04756/) (2020)
- [1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed](https://mlsanity.com/paper/2102.02888/) (2021)
- [Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2102.04487/) (2021)
- [All You Need is a Good Functional Prior for Bayesian Deep Learning](https://mlsanity.com/paper/2011.12829/) (2020)
- [Blockchain Assisted Decentralized Federated Learning (BLADE-FL): Performance Analysis and Resource Allocation](https://mlsanity.com/paper/2101.06905/) (2021)
- [Communication-Efficient Adaptive Federated Learning](https://mlsanity.com/paper/2205.02719/) (2022)
- [Federated Learning for Internet of Things: Applications, Challenges, and Opportunities](https://mlsanity.com/paper/2111.07494/) (2021)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [Rethinking gradient sparsification as total error minimization](https://mlsanity.com/paper/2108.00951/) (2021)
- [A Survey on Large-scale Machine Learning](https://mlsanity.com/paper/2008.03911/) (2020)
- [Communication-Efficient Federated Learning with Compensated Overlap-FedAvg](https://mlsanity.com/paper/2012.06706/) (2020)
- [Federated Learning for Privacy Preservation in Smart Healthcare Systems: A Comprehensive Survey](https://mlsanity.com/paper/2203.09702/) (2022)
- [Quantized Adam with Error Feedback](https://mlsanity.com/paper/2004.14180/) (2020)
- [Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimization](https://mlsanity.com/paper/2008.01558/) (2020)
- [Time-Correlated Sparsification for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2101.08837/) (2021)
- [Adaptive Federated Learning With Gradient Compression in Uplink NOMA](https://mlsanity.com/paper/2003.01344/) (2020)
- [Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theory](https://mlsanity.com/paper/2201.02664/) (2022)
- [Low-latency Federated Learning with DNN Partition in Distributed Industrial IoT Networks](https://mlsanity.com/paper/2210.14700/) (2022)
- [Preserving Privacy and Security in Federated Learning](https://mlsanity.com/paper/2202.03402/) (2022)
- [Federated Learning with Nesterov Accelerated Gradient](https://mlsanity.com/paper/2009.08716/) (2020)
- [Efficient-Adam: Communication-Efficient Distributed Adam](https://mlsanity.com/paper/2205.14473/) (2022)
- [CADA: Communication-Adaptive Distributed Adam](https://mlsanity.com/paper/2012.15469/) (2020)
- [Large-Scale Deep Learning Optimizations: A Comprehensive Survey](https://mlsanity.com/paper/2111.00856/) (2021)
- [Non-Convergence and Limit Cycles in the Adam optimizer](https://mlsanity.com/paper/2210.02070/) (2022)
- [Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees](https://mlsanity.com/paper/1911.08727/) (2019)
- [Detached Error Feedback for Distributed SGD with Random Sparsification](https://mlsanity.com/paper/2004.05298/) (2020)
- [Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacy](https://mlsanity.com/paper/2304.04164/) (2023)

## Built on by (most cited first, 1 of 1)

- [Closing the Generalization Gap in Parameter-efficient Federated Edge Learning](https://mlsanity.com/paper/2511.23282/) (2025)

## Abstract (arXiv)

Adaptive moment estimation (Adam), as a Stochastic Gradient Descent (SGD) variant, has gained widespread popularity in federated learning (FL) due to its fast convergence. However, federated Adam (FedAdam) algorithms suffer from a threefold increase in uplink communication overhead compared to federated SGD (FedSGD) algorithms, which arises from the necessity to transmit both local model updates and first and second moment estimates from distributed devices to the centralized server for aggregation. Driven by this issue, we propose a novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices sparsify the updates of local model parameters and moment estimates and subsequently upload the sparse representations to the centralized server. To further reduce the communication overhead, the updates of local model parameters and moment estimates incorporate a shared sparse mask (SSM) into the sparsification process, eliminating the need for three separate sparse masks. Theoretically, we develop an upper bound on the divergence between the local model trained by FedAdam-SSM and the desired model trained by centralized Adam, which is related to sparsification error and imbalanced data distribution. By minimizing the divergence bound between the model trained by FedAdam-SSM and centralized Adam, we optimize the SSM to mitigate the learning performance degradation caused by sparsification error. Additionally, we provide convergence bounds for FedAdam-SSM in both convex and non-convex objective function settings, and investigate the impact of local epoch, learning rate and sparsification ratio on the convergence rate of FedAdam-SSM. Experimental results show that FedAdam-SSM outperforms baselines in terms of convergence rate (over 1.1× faster than the sparse FedAdam baselines) and test accuracy (over 14.5% ahead of the quantized FedAdam baselines).