# GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training

arXiv: https://arxiv.org/abs/1811.03617 · HTML: https://mlsanity.com/paper/1811.03617/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Nov 2018.

## Built on by (most cited first, 21 of 21)

- [PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication](https://mlsanity.com/paper/2203.10428/) (2022)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training](https://mlsanity.com/paper/2104.11125/) (2021)
- [Distributed Learning Systems with First-order Methods](https://mlsanity.com/paper/2104.05245/) (2021)
- [Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks](https://mlsanity.com/paper/2104.07085/) (2021)
- [Block Walsh-Hadamard Transform Based Binary Layers in Deep Neural Networks](https://mlsanity.com/paper/2201.02711/) (2022)
- [LASER: Linear Compression in Wireless Distributed Optimization](https://mlsanity.com/paper/2310.13033/) (2023)
- [Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression](https://mlsanity.com/paper/2204.06787/) (2022)
- [FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantization](https://mlsanity.com/paper/2406.18156/) (2024)
- [Reconciling Security and Communication Efficiency in Federated Learning](https://mlsanity.com/paper/2207.12779/) (2022)
- [Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression](https://mlsanity.com/paper/2211.14292/) (2022)
- [Compressing gradients by exploiting temporal correlation in momentum-SGD](https://mlsanity.com/paper/2108.07827/) (2021)
- [Peering Beyond the Gradient Veil with Distributed Auto Differentiation](https://mlsanity.com/paper/2102.09631/) (2021)
- [Communication-Efficient Federated Learning by Exploiting Spatio-Temporal Correlations of Gradients](https://mlsanity.com/paper/2601.10491/) (2026)
- [Unbiased Single-scale and Multi-scale Quantizers for Distributed Optimization](https://mlsanity.com/paper/2109.12497/) (2021)
- [Embedding Compression for Text Classification Using Dictionary Screening](https://mlsanity.com/paper/2211.12715/) (2022)
- [Quantize Once, Train Fast: Allreduce-Compatible Compression with Provable Guarantees](https://mlsanity.com/paper/2305.18627/) (2023)
- [DEFT: Exploiting Gradient Norm Difference between Model Layers for Scalable Gradient Sparsification](https://mlsanity.com/paper/2307.03500/) (2023)
- [Contractive error feedback for gradient compression](https://mlsanity.com/paper/2312.08538/) (2023)
- [RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient Compression on Remote Sensing Image Interpretation](https://mlsanity.com/paper/2312.17530/) (2023)
- [Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations](https://mlsanity.com/paper/2604.14751/) (2026)

## Abstract (arXiv)

Data parallelism can boost the training speed of convolutional neural networks (CNN), but could suffer from significant communication costs caused by gradient aggregation. To alleviate this problem, several scalar quantization techniques have been developed to compress the gradients. But these techniques could perform poorly when used together with decentralized aggregation protocols like ring all-reduce (RAR), mainly due to their inability to directly aggregate compressed gradients. In this paper, we empirically demonstrate the strong linear correlations between CNN gradients, and propose a gradient vector quantization technique, named GradiVeQ, to exploit these correlations through principal component analysis (PCA) for substantial gradient dimension reduction. GradiVeQ enables direct aggregation of compressed gradients, hence allows us to build a distributed learning system that parallelizes GradiVeQ gradient compression and RAR communications. Extensive experiments on popular CNNs demonstrate that applying GradiVeQ slashes the wall-clock gradient aggregation time of the original RAR by more than 5X without noticeable accuracy loss, and reduces the end-to-end training time by almost 50%. The results also show that GradiVeQ is compatible with scalar quantization techniques such as QSGD (Quantized SGD), and achieves a much higher speed-up gain under the same compression ratio.