# Quantizing data for distributed learning

arXiv: https://arxiv.org/abs/2012.07913 · HTML: https://mlsanity.com/paper/2012.07913/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2020.

## Built on by (most cited first, 8 of 8)

- [Quantizing Heavy-tailed Data in Statistical Estimation: (Near) Minimax Rates, Covariate Quantization, and Uniform Recovery](https://mlsanity.com/paper/2212.14562/) (2022)
- [Power Allocation for Wireless Federated Learning using Graph Neural Networks](https://mlsanity.com/paper/2111.07480/) (2021)
- [Goal-Oriented Quantization: Analysis, Design, and Application to Resource Allocation](https://mlsanity.com/paper/2209.15347/) (2022)
- [Quantized Low-Rank Multivariate Regression with Random Dithering](https://mlsanity.com/paper/2302.11197/) (2023)
- [A Parameter-Free Two-Bit Covariance Estimator with Improved Operator Norm Error Rate](https://mlsanity.com/paper/2308.16059/) (2023)
- [Learning to Transmit with Provable Guarantees in Wireless Federated Learning](https://mlsanity.com/paper/2304.09329/) (2023)
- [Learning in Distributed Contextual Linear Bandits Without Sharing the Context](https://mlsanity.com/paper/2206.04180/) (2022)
- [Machine Learning and CPU (Central Processing Unit) Scheduling Co-Optimization over a Network of Computing Centers](https://mlsanity.com/paper/2510.25176/) (2025)

## Abstract (arXiv)

We consider machine learning applications that train a model by leveraging data distributed over a trusted network, where communication constraints can create a performance bottleneck. A number of recent approaches propose to overcome this bottleneck through compression of gradient updates. However, as models become larger, so does the size of the gradient updates. In this paper, we propose an alternate approach to learn from distributed data that quantizes data instead of gradients, and can support learning over applications where the size of gradient updates is prohibitive. Our approach leverages the dependency of the computed gradient on data samples, which lie in a much smaller space in order to perform the quantization in the smaller dimension data space. At the cost of an extra gradient computation, the gradient estimate can be refined by conveying the difference between the gradient at the quantized data point and the original gradient using a small number of bits. Lastly, in order to save communication, our approach adds a layer that decides whether to transmit a quantized data sample or not based on its importance for learning. We analyze the convergence of the proposed approach for smooth convex and non-convex objective functions and show that we can achieve order optimal convergence rates with communication that mostly depends on the data rather than the model (gradient) dimension. We use our proposed algorithm to train ResNet models on the CIFAR-10 and ImageNet datasets, and show that we can achieve an order of magnitude savings over gradient compression methods. These communication savings come at the cost of increasing computation at the learning agent, and thus our approach is beneficial in scenarios where communication load is the main problem.