# textttDeepSqueeze: Decentralization Meets Error-Compensated Compression

arXiv: https://arxiv.org/abs/1907.07346 · HTML: https://mlsanity.com/paper/1907.07346/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2019.

## Built on by (most cited first, 22 of 22)

- [An Improved Analysis of Gradient Tracking for Decentralized Machine Learning](https://mlsanity.com/paper/2202.03836/) (2022)
- [BEER: Fast O(1/T) Rate for Decentralized Nonconvex Optimization with Communication Compression](https://mlsanity.com/paper/2201.13320/) (2022)
- [Decentralized Local Stochastic Extra-Gradient for Variational Inequalities](https://mlsanity.com/paper/2106.08315/) (2021)
- [On the Convergence of Distributed Stochastic Bilevel Optimization Algorithms over a Network](https://mlsanity.com/paper/2206.15025/) (2022)
- [Compressed Gradient Tracking for Decentralized Optimization Over General Directed Networks](https://mlsanity.com/paper/2106.07243/) (2021)
- [CEDAS: A Compressed Decentralized Stochastic Gradient Method with Improved Convergence](https://mlsanity.com/paper/2301.05872/) (2023)
- [Convergence and Privacy of Decentralized Nonconvex Optimization with Gradient Clipping and Communication Compression](https://mlsanity.com/paper/2305.09896/) (2023)
- [Sparse-Push: Communication- & Energy-Efficient Decentralized Distributed Learning over Directed & Time-Varying Graphs with non-IID Datasets](https://mlsanity.com/paper/2102.05715/) (2021)
- [CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning](https://mlsanity.com/paper/2106.07155/) (2021)
- [Decentralized Composite Optimization with Compression](https://mlsanity.com/paper/2108.04448/) (2021)
- [Communication-Efficient Design for Quantized Decentralized Federated Learning](https://mlsanity.com/paper/2303.08423/) (2023)
- [Neighborhood Gradient Clustering: An Efficient Decentralized Learning Method for Non-IID Data Distributions](https://mlsanity.com/paper/2209.14390/) (2022)
- [Towards Faster Decentralized Stochastic Optimization with Communication Compression](https://mlsanity.com/paper/2405.20114/) (2024)
- [Near-Optimal Decentralized Momentum Method for Nonconvex-PL Minimax Problems](https://mlsanity.com/paper/2304.10902/) (2023)
- [Differential error feedback for communication-efficient decentralized learning](https://mlsanity.com/paper/2406.18418/) (2024)
- [Low Precision Decentralized Distributed Training over IID and non-IID Data](https://mlsanity.com/paper/2111.09389/) (2021)
- [Compressed Decentralized Momentum Stochastic Gradient Methods for Nonconvex Optimization](https://mlsanity.com/paper/2508.04950/) (2025)
- [Improved Convergence in Parameter-Agnostic Error Feedback through Momentum](https://mlsanity.com/paper/2511.14501/) (2025)
- [Averaging Rate Scheduler for Decentralized Learning on Heterogeneous Data](https://mlsanity.com/paper/2403.03292/) (2024)
- [AdaGossip: Adaptive Consensus Step-size for Decentralized Deep Learning with Communication Compression](https://mlsanity.com/paper/2404.05919/) (2024)
- [SeedFlood: A Step Toward Scalable Decentralized Fine-Tuning of LLMs](https://mlsanity.com/paper/2602.18181/) (2026)
- [Achieving Linear Speedup in Decentralized Stochastic Compositional Minimax Optimization](https://mlsanity.com/paper/2307.13430/) (2023)

## Abstract (arXiv)

Communication is a key bottleneck in distributed training. Recently, an error-compensated compression technology was particularly designed for the centralized learning and receives huge successes, by showing significant advantages over state-of-the-art compression based methods in saving the communication cost. Since the decentralized training has been witnessed to be superior to the traditional centralized training in the communication restricted scenario, therefore a natural question to ask is "how to apply the error-compensated technology to the decentralized learning to further reduce the communication cost." However, a trivial extension of compression based centralized training algorithms does not exist for the decentralized scenario. key difference between centralized and decentralized training makes this extension extremely non-trivial. In this paper, we propose an elegant algorithmic design to employ error-compensated stochastic gradient descent for the decentralized scenario, named textttDeepSqueeze. Both the theoretical analysis and the empirical study are provided to show the proposed textttDeepSqueeze algorithm outperforms the existing compression based decentralized learning algorithms. To the best of our knowledge, this is the first time to apply the error-compensated compression to the decentralized learning.