# CSER: Communication-efficient SGD with Error Reset

arXiv: https://arxiv.org/abs/2007.13221 · HTML: https://mlsanity.com/paper/2007.13221/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Jul 2020.

## Built on by (most cited first, 27 of 27)

- [A Field Guide to Federated Optimization](https://mlsanity.com/paper/2107.06917/) (2021)
- [EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback](https://mlsanity.com/paper/2106.05203/) (2021)
- [1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed](https://mlsanity.com/paper/2102.02888/) (2021)
- [Rethinking gradient sparsification as total error minimization](https://mlsanity.com/paper/2108.00951/) (2021)
- [Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression](https://mlsanity.com/paper/2206.03665/) (2022)
- [Momentum Provably Improves Error Feedback!](https://mlsanity.com/paper/2305.15155/) (2023)
- [Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees](https://mlsanity.com/paper/2206.01299/) (2022)
- [Time-Correlated Sparsification for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2101.08837/) (2021)
- [1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed](https://mlsanity.com/paper/2104.06069/) (2021)
- [lo-fi: distributed fine-tuning without communication](https://mlsanity.com/paper/2210.11948/) (2022)
- [Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression](https://mlsanity.com/paper/2305.07612/) (2023)
- [Efficient Asynchronous Federated Learning with Sparsification and Quantization](https://mlsanity.com/paper/2312.15186/) (2023)
- [EF21 with Bells & Whistles: Six Algorithmic Extensions of Modern Error Feedback](https://mlsanity.com/paper/2110.03294/) (2021)
- [On Faster Convergence of Scaled Sign Gradient Descent](https://mlsanity.com/paper/2109.01806/) (2021)
- [Distributed Online Learning for Joint Regret with Communication Constraints](https://mlsanity.com/paper/2102.07521/) (2021)
- [Compressed Communication for Distributed Training: Adaptive Methods and System](https://mlsanity.com/paper/2105.07829/) (2021)
- [Ordered Momentum for Asynchronous SGD](https://mlsanity.com/paper/2407.19234/) (2024)
- [FedREP: A Byzantine-Robust, Communication-Efficient and Privacy-Preserving Framework for Federated Learning](https://mlsanity.com/paper/2303.05206/) (2023)
- [Escaping Saddle Points in Heterogeneous Federated Learning via Distributed SGD with Communication Compression](https://mlsanity.com/paper/2310.19059/) (2023)
- [Communication-Efficient Distributed Learning with Local Immediate Error Compensation](https://mlsanity.com/paper/2402.11857/) (2024)
- [Communication-Efficient Adam-Type Algorithms for Distributed Data Mining](https://mlsanity.com/paper/2210.07454/) (2022)
- [Sketched Adaptive Federated Deep Learning: A Sharp Convergence Analysis](https://mlsanity.com/paper/2411.06770/) (2024)
- [Distributed Sign Momentum with Local Steps for Training Transformers](https://mlsanity.com/paper/2411.17866/) (2024)
- [Quantize Once, Train Fast: Allreduce-Compatible Compression with Provable Guarantees](https://mlsanity.com/paper/2305.18627/) (2023)
- [Contractive error feedback for gradient compression](https://mlsanity.com/paper/2312.08538/) (2023)
- [RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient Compression on Remote Sensing Image Interpretation](https://mlsanity.com/paper/2312.17530/) (2023)
- [SA-PEF: Step-Ahead Partial Error Feedback for Efficient Federated Learning](https://mlsanity.com/paper/2601.20738/) (2026)

## Abstract (arXiv)

The scalability of Distributed Stochastic Gradient Descent (SGD) is today limited by communication bottlenecks. We propose a novel SGD variant: Communication-efficient SGD with Error Reset, or CSER. The key idea in CSER is first a new technique called "error reset" that adapts arbitrary compressors for SGD, producing bifurcated local models with periodic reset of resulting local residual errors. Second we introduce partial synchronization for both the gradients and the models, leveraging advantages from them. We prove the convergence of CSER for smooth non-convex problems. Empirical results show that when combined with highly aggressive compressors, the CSER algorithms accelerate the distributed training by nearly 10x for CIFAR-100, and by 4.5x for ImageNet.