mlsanityData to 14 Oct 2026

Magnitude Matters: Fixing SIGNSGD Through Magnitude-Aware Sparsification in the Presence of Data Heterogeneity

arXiv 2302.09634 ↗First posted 19 Feb 2023cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Communication overhead has become one of the major bottlenecks in the distributed training of deep neural networks. To alleviate the concern, various gradient compression methods have been proposed, and sign-based algorithms are of surging interest. However, SIGNSGD fails to converge in the presence of data heterogeneity, which is commonly observed in the emerging federated learning (FL) paradigm. Error feedback has been proposed to address the non-convergence issue. Nonetheless, it requires the workers to locally keep track of the compression errors, which renders it not suitable for FL since the workers may not participate in the training throughout the learning process. In this paper, we propose a magnitude-driven sparsification scheme, which addresses the non-convergence issue of SIGNSGD while further improving communication efficiency. Moreover, the local update scheme is further incorporated to improve the learning performance, and the convergence of the proposed method is established. The effectiveness of the proposed scheme is validated through experiments on Fashion-MNIST, CIFAR-10, and CIFAR-100 datasets.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 36 of its 46 references that are in the map. First posted on arXiv 19 Feb 2023.

Builds on

36 of 46 references.

References · 36

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  3. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  4. 2018Local SGD Converges Fast and Communicates Littlecited by 605
  5. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  6. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  7. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  8. 2018Sparsified SGD with Memorycited by 376
  9. 2019Error Feedback Fixes SignSGD and other Gradient Compression Schemescited by 310
  10. 2017TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learningcited by 256
  11. 2018The Convergence of Sparsified Gradient Methodscited by 229
  12. 2017Gradient Sparsification for Communication-Efficient Distributed Optimizationcited by 201
24 more
  1. 2019Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computationscited by 181
  2. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  3. 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 160
  4. 2017Revisiting Distributed Synchronous SGDcited by 142
  5. 2020Understanding Gradient Clipping in Private SGD: A Geometric Perspectivecited by 142
  6. 2018ATOMO: Communication-efficient Learning via Atomic Sparsificationcited by 136
  7. 2019Enabling Spike-based Backpropagation for Training Deep Neural Network Architecturescited by 123
  8. 2019Distributed Learning with Compressed Gradient Differencescited by 112
  9. 2020UVeQFed: Universal Vector Quantization for Federated Learningcited by 107
  10. 2019DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compressioncited by 99
  11. 2020Acceleration for Compressed Gradient Descent in Distributed and Federated Optimizationcited by 90
  12. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  13. 2021Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacycited by 69
  14. 2019Stochastic Distributed Learning with Gradient Quantization and Variance Reductioncited by 65
  15. 2018Sparse Binary Compression: Towards Distributed Deep Learning with minimal Communicationcited by 51
  16. 2019Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedbackcited by 50
  17. 2019Stochastic Sign Descent Methods: New Algorithms and Better Theorycited by 49
  18. 2020Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guaranteescited by 46
  19. 2020A Better Alternative to Error Feedback for Communication-Efficient Distributed Learningcited by 43
  20. 2021Rethinking gradient sparsification as total error minimizationcited by 41
  21. 2020Stochastic-Sign SGD for Federated Learning with Theoretical Guaranteescited by 35
  22. 2019A Double Residual Compression Algorithm for Efficient Distributed Learningcited by 31
  23. 2019Distributed Training with Heterogeneous Data: Bridging Median- and Mean-Based Algorithmscited by 29
  24. 2018signSGD with Majority Vote is Communication Efficient And Fault Tolerantcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 0; 2026: 1.

2 papers.

  1. 2023Convergence of Sign-based Random Reshuffling Algorithms for Nonconvex Optimizationcited by 2
  2. 2026Collaborative Compressors in Distributed Mean Estimation with Limited Communication Budgetcited by 1