# Convert, compress, correct: Three steps toward communication-efficient DNN training

arXiv: https://arxiv.org/abs/2203.09044 · HTML: https://mlsanity.com/paper/2203.09044/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 8 of its 17 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 8 of 8)

- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Sparsified SGD with Memory](https://mlsanity.com/paper/1809.07599/) (2018)
- [Error Feedback Fixes SignSGD and other Gradient Compression Schemes](https://mlsanity.com/paper/1901.09847/) (2019)
- [The Convergence of Sparsified Gradient Methods](https://mlsanity.com/paper/1809.10505/) (2018)
- [ATOMO: Communication-efficient Learning via Atomic Sparsification](https://mlsanity.com/paper/1806.04090/) (2018)
- [Decentralized optimization over noisy, rate-constrained networks: Achieving consensus by communicating differences](https://mlsanity.com/paper/2010.11292/) (2020)
- [DNN gradient lossless compression: Can GenNorm be the answer?](https://mlsanity.com/paper/2111.07599/) (2021)
- [Lossy Gradient Compression: How Much Accuracy Can One Bit Buy?](https://mlsanity.com/paper/2202.02812/) (2022)

## Abstract (arXiv)

In this paper, we introduce a novel algorithm, mathsfCO3, for communication-efficiency distributed Deep Neural Network (DNN) training. mathsfCO3 is a joint training/communication protocol, which encompasses three processing steps for the network gradients: (i) quantization through floating-point conversion, (ii) lossless compression, and (iii) error correction. These three components are crucial in the implementation of distributed DNN training over rate-constrained links. The interplay of these three steps in processing the DNN gradients is carefully balanced to yield a robust and high-performance scheme. The performance of the proposed scheme is investigated through numerical evaluations over CIFAR-10.