mlsanityData to 7 Oct 2026

Error Feedback Fixes SignSGD and other Gradient Compression Schemes

arXiv 1901.09847 ↗First posted 28 Jan 2019cs.LG math.OC stat.MLcited by 306

Not on the map. Why?

Abstract (arXiv)

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD does not converge to the optimum. Further, even when it does converge, signSGD may generalize poorly when compared with SGD. These issues arise because of the biased nature of the sign compression operator. We then show that using error-feedback, i.e. incorporating the error made by the compression operator into the next step, overcomes these issues. We prove that our algorithm EF-SGD with arbitrary compression operator achieves the same rate of convergence as SGD without any additional assumptions. Thus EF-SGD achieves gradient compression for free. Our experiments thoroughly substantiate the theory and show that error-feedback improves both convergence and generalization. Code can be found at urlhttps://github.com/epfml/error-feedback-SGD.

Key facts. Cited by 306 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 74; 2022: 40; 2023: 47; 2024: 45; 2025: 55; 2026: 44.

306 papers.

Sort:

By area:

  1. 2021A Field Guide to Federated Optimizationcited by 312
  2. 2021The Values Encoded in Machine Learning Researchcited by 172
  3. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  4. 2021EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedbackcited by 131
  5. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
  6. 2021Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradientscited by 122
  7. 2023A Comprehensive Survey of Dataset Distillationcited by 114
  8. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
  9. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87
  10. 2021MARINA: Faster Non-Convex Distributed Learning with Compressioncited by 77
  11. 2025Muon Optimizes Under Spectral Norm Constraintscited by 77
  12. 2023FedExP: Speeding Up Federated Averaging via Extrapolationcited by 68
38 more
  1. 2021From Distributed Machine Learning to Federated Learning: A Surveycited by 65
  2. 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
  3. 2022Communication-Efficient Adaptive Federated Learningcited by 55
  4. 2025ASGO: Adaptive Structured Gradient Optimizationcited by 55
  5. 2021EDEN: Communication-Efficient and Robust Distributed Mean Estimation for Federated Learningcited by 53
  6. 2021DRIVE: One-bit Distributed Mean Estimationcited by 52
  7. 2021FedNL: Making Newton-Type Methods Applicable to Federated Learningcited by 50
  8. 2024From Gradient Clipping to Normalization for Heavy Tailed SGDcited by 48
  9. 2021SparseFed: Mitigating Model Poisoning Attacks in Federated Learning with Sparsificationcited by 46
  10. 2021Rethinking gradient sparsification as total error minimizationcited by 41
  11. 2022BEER: Fast O(1/T) Rate for Decentralized Nonconvex Optimization with Communication Compressioncited by 41
  12. 2022A Survey on Gradient Inversion: Attacks, Defenses and Future Directionscited by 39
  13. 2023A Guide Through the Zoo of Biased SGDcited by 39
  14. 2025Lions and Muons: Optimization via Stochastic Frank-Wolfe under Heavy-Tailed Noisecited by 38
  15. 2022Stochastic Gradient Descent-Ascent: Unified Theory and New Efficient Methodscited by 37
  16. 2021Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devicescited by 36
  17. 2022Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Datacited by 36
  18. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  19. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  20. 2022Joint Privacy Enhancement and Quantization in Federated Learningcited by 34
  21. 2022Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compressioncited by 33
  22. 2021An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systemscited by 32
  23. 2021Cross-Gradient Aggregation for Decentralized Learning from Non-IID datacited by 32
  24. 2021Asynchronous Stochastic Optimization Robust to Arbitrary Delayscited by 32
  25. 2021QuPeD: Quantized Personalization via Distillation with Applications to Federated Learningcited by 32
  26. 2023Momentum Provably Improves Error Feedback!cited by 31
  27. 2021Byzantine-robust Federated Learning through Collaborative Malicious Gradient Filteringcited by 28
  28. 2021ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Trainingcited by 27
  29. 20223PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregationcited by 27
  30. 2023DoCoFL: Downlink Compression for Cross-Device Federated Learningcited by 27
  31. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  32. 2021Time-Correlated Sparsification for Communication-Efficient Federated Learningcited by 25
  33. 2023Two Sides of One Coin: the Limits of Untuned SGD and the Power of Adaptive Methodscited by 23
  34. 2021ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Trainingcited by 22
  35. 2021Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimizationcited by 21
  36. 2021Learned Gradient Compression for Distributed Deep Learningcited by 20
  37. 2021Robust Training in High Dimensions via Block Coordinate Geometric Median Descentcited by 20
  38. 2021Lower Bounds and Optimal Algorithms for Smooth and Strongly Convex Decentralized Optimization Over Time-Varying Networkscited by 19

50 most cited of 306 All 306 (JSON)