Data to 7 Oct 2026

signSGD: Compressed Optimisation for Non-Convex Problems

arXiv 1802.04434 ↗First posted 13 Feb 2018cs.LG cs.DC math.OCcited by 561

Not on the map. Why?

Abstract (arXiv)

Training large neural networks requires distributing learning across multiple workers, where the cost of communicating gradients can be a significant bottleneck. signSGD alleviates this problem by transmitting just the sign of each minibatch stochastic gradient. We prove that it can get the best of both worlds: compressed gradients and SGD-level convergence rate. The relative ℓ1/ℓ2 geometry of gradients, noise and curvature informs whether signSGD or SGD is theoretically better suited to a particular problem. On the practical side we find that the momentum counterpart of signSGD is able to match the accuracy and convergence speed of Adam on deep Imagenet models. We extend our theory to the distributed setting, where the parameter server uses majority vote to aggregate gradient signs from each worker enabling 1-bit compression of worker-server communication in both directions. Using a theorem by Gauss we prove that majority vote can achieve the same reduction in variance as full precision distributed SGD. Thus, there is great promise for sign-based optimisation schemes to achieve fast communication and fast convergence. Code to reproduce experiments is to be found at https://github.com/jxbz/signSGD .

Key facts. Cited by 561 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 96; 2022: 78; 2023: 88; 2024: 98; 2025: 90; 2026: 111.

561 papers.

Sort:

By area:

2023Symbolic Discovery of Optimization Algorithmscited by 423
2021A Field Guide to Federated Optimizationcited by 312
2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 260
2025Training Deep Learning Models with Norm-Constrained LMOscited by 177
2024Old Optimizer, New Norm: An Anthologycited by 173
2021FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learningcited by 140
2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
2023A Comprehensive Survey of Dataset Distillationcited by 114
2023Noise Is Not the Main Factor Behind the Gap Between SGD and Adam on Transformers, but Sign Descent Might Becited by 114
2023A Spectral Condition for Feature Learningcited by 101
2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100

38 more

2021Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applicationscited by 96
2022Automatic Clipping: Differentially Private Deep Learning Made Easier and Strongercited by 87
2022Survey on Federated Learning Threats: concepts, taxonomy on attacks and defences, experimental study and challengescited by 81
2022Robustness to Unbounded Smoothness of Generalized SignSGDcited by 80
2025Muon Optimizes Under Spectral Norm Constraintscited by 77
2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
2022Asynchronous SGD Beats Minibatch SGD Under Arbitrary Delayscited by 61
2021Provably Secure Federated Learning against Malicious Clientscited by 60
2022Communication-Efficient Adaptive Federated Learningcited by 55
2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
2021FedLab: A Flexible Federated Learning Frameworkcited by 54
2024Implicit Bias of AdamW: ℓ_∞ Norm Constrained Optimizationcited by 54
2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 53
2021Federated Learning over Wireless Device-to-Device Networks: Algorithms and Convergence Analysiscited by 51
2023Toward Understanding Why Adam Converges Faster Than SGD for Transformerscited by 50
2023Lion Secretly Solves Constrained Optimization: As Lyapunov Predictscited by 50
2021Federated Learning: A Signal Processing Perspectivecited by 49
2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
2021Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularizationcited by 48
2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
2021GradInit: Learning to Initialize Neural Networks for Stable and Efficient Trainingcited by 44
2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
2024Cautious Optimizers: Improving Training with One Line of Codecited by 42
2024Deconstructing What Makes a Good Optimizer for Language Modelscited by 40
2025In Search of Adam's Secret Saucecited by 40
2025Weight-sparse transformers have interpretable circuitscited by 40
2022The Fundamental Price of Secure Aggregation in Differentially Private Federated Learningcited by 39
2023An Introduction to Bi-level Optimization: Foundations and Applications in Signal Processing and Machine Learningcited by 39
2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
2022Gradient Obfuscation Gives a False Sense of Security in Federated Learningcited by 37
2022Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Datacited by 36
2022Joint Privacy Enhancement and Quantization in Federated Learningcited by 34
2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
2025Implicit Bias of Spectral Descent and Muon on Multiclass Separable Datacited by 33
2022Federated Learning with Sparsified Model Perturbation: Improving Accuracy under Client-Level Differential Privacycited by 32
2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32

50 most cited of 561 All 561 (JSON)