# ErrorCompensatedX: error compensation for variance reduced algorithms

arXiv: https://arxiv.org/abs/2108.02102 · HTML: https://mlsanity.com/paper/2108.02102/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Aug 2021.

## Built on by (most cited first, 5 of 5)

- [Error Feedback under (L0,L1)-Smoothness: Normalization and Momentum](https://mlsanity.com/paper/2410.16871/) (2024)
- [DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification](https://mlsanity.com/paper/2307.03500/) (2023)
- [MiCRO: Near-Zero Cost Gradient Sparsification for Scaling and Accelerating Distributed DNN Training](https://mlsanity.com/paper/2310.00967/) (2023)
- [Preserving Near-Optimal Gradient Sparsification Cost for Scalable Distributed Deep Learning](https://mlsanity.com/paper/2402.13781/) (2024)
- [A Tight Theory of Error Feedback Algorithms in Distributed Optimization](https://mlsanity.com/paper/2605.31594/) (2026)

## Abstract (arXiv)

Communication cost is one major bottleneck for the scalability for distributed learning. One approach to reduce the communication cost is to compress the gradient during communication. However, directly compressing the gradient decelerates the convergence speed, and the resulting algorithm may diverge for biased compression. Recent work addressed this problem for stochastic gradient descent by adding back the compression error from the previous step. This idea was further extended to one class of variance reduced algorithms, where the variance of the stochastic gradient is reduced by taking a moving average over all history gradients. However, our analysis shows that just adding the previous step's compression error, as done in existing work, does not fully compensate the compression error. So, we propose ErrorCompensatedX, which uses the compression error from the previous two steps. We show that ErrorCompensatedX can achieve the same asymptotic convergence rate with the training without compression. Moreover, we provide a unified theoretical analysis framework for this class of variance reduced algorithms, with or without error compensation.