mlsanityData to 14 Oct 2026

Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization

arXiv 2405.17932 ↗First posted 28 May 2024cs.LG cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

Adaptive moment estimation (Adam), as a Stochastic Gradient Descent (SGD) variant, has gained widespread popularity in federated learning (FL) due to its fast convergence. However, federated Adam (FedAdam) algorithms suffer from a threefold increase in uplink communication overhead compared to federated SGD (FedSGD) algorithms, which arises from the necessity to transmit both local model updates and first and second moment estimates from distributed devices to the centralized server for aggregation. Driven by this issue, we propose a novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices sparsify the updates of local model parameters and moment estimates and subsequently upload the sparse representations to the centralized server. To further reduce the communication overhead, the updates of local model parameters and moment estimates incorporate a shared sparse mask (SSM) into the sparsification process, eliminating the need for three separate sparse masks. Theoretically, we develop an upper bound on the divergence between the local model trained by FedAdam-SSM and the desired model trained by centralized Adam, which is related to sparsification error and imbalanced data distribution. By minimizing the divergence bound between the model trained by FedAdam-SSM and centralized Adam, we optimize the SSM to mitigate the learning performance degradation caused by sparsification error. Additionally, we provide convergence bounds for FedAdam-SSM in both convex and non-convex objective function settings, and investigate the impact of local epoch, learning rate and sparsification ratio on the convergence rate of FedAdam-SSM. Experimental results show that FedAdam-SSM outperforms baselines in terms of convergence rate (over 1.1× faster than the sparse FedAdam baselines) and test accuracy (over 14.5% ahead of the quantized FedAdam baselines).

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 63 references that are in the map. First posted on arXiv 28 May 2024.

Builds on

31 of 63 references.

References · 31

  1. 2020Federated Learning with Matched Averagingcited by 564
  2. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  3. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 387
  4. 2020Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approachcited by 81
  5. 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speedcited by 75
  6. 2021Adaptive Quantization of Model Updates for Communication-Efficient Federated Learningcited by 63
  7. 2020All You Need is a Good Functional Prior for Bayesian Deep Learningcited by 61
  8. 2021Blockchain Assisted Decentralized Federated Learning (BLADE-FL): Performance Analysis and Resource Allocationcited by 56
  9. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  10. 2021Federated Learning for Internet of Things: Applications, Challenges, and Opportunitiescited by 50
  11. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  12. 2021Rethinking gradient sparsification as total error minimizationcited by 41
19 more
  1. 2020A Survey on Large-scale Machine Learningcited by 35
  2. 2020Communication-Efficient Federated Learning with Compensated Overlap-FedAvgcited by 31
  3. 2022Federated Learning for Privacy Preservation in Smart Healthcare Systems: A Comprehensive Surveycited by 31
  4. 2020Quantized Adam with Error Feedbackcited by 27
  5. 2020Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimizationcited by 26
  6. 2021Time-Correlated Sparsification for Communication-Efficient Federated Learningcited by 26
  7. 2020Adaptive Federated Learning With Gradient Compression in Uplink NOMAcited by 19
  8. 2022Optimizing the Communication-Accuracy Trade-off in Federated Learning with Rate-Distortion Theorycited by 14
  9. 2022Low-latency Federated Learning with DNN Partition in Distributed Industrial IoT Networkscited by 14
  10. 2022Preserving Privacy and Security in Federated Learningcited by 13
  11. 2020Federated Learning with Nesterov Accelerated Gradientcited by 11
  12. 2022Efficient-Adam: Communication-Efficient Distributed Adamcited by 11
  13. 2020CADA: Communication-Adaptive Distributed Adamcited by 10
  14. 2021Large-Scale Deep Learning Optimizations: A Comprehensive Surveycited by 10
  15. 2022Non-Convergence and Limit Cycles in the Adam optimizercited by 9
  16. 2019Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guaranteescited by 8
  17. 2020Detached Error Feedback for Distributed SGD with Random Sparsificationcited by 6
  18. 2023Gradient Sparsification for Efficient Wireless Federated Learning with Differential Privacycited by 3
  19. 2022Communication-Efficient Adam-Type Algorithms for Distributed Data Miningcited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Closing the Generalization Gap in Parameter-efficient Federated Edge Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Communication-Efficient Federated Learning via Regularized Sparse Random Networks
  2. 2024FedComLoc: Communication-Efficient Distributed Training of Sparse and Quantized Models
  3. 2022Sparse Random Networks for Communication-Efficient Federated Learning
  4. 2024Masked Random Noise for Communication Efficient Federated Learning
  5. 2024Adaptive Federated Learning Over the Air
  6. 2026FedSGM: A Unified Framework for Constraint Aware, Bidirectionally Compressed, Multi-Step Federated Optimization
  7. 2026Q-LocalAdam: Memory-Efficient Client-Side Adaptive Optimization for Edge Federated Learning
  8. 2025FedSparQ: Adaptive Sparse Quantization with Error Feedback for Robust & Efficient Federated Learning