# Double Quantization for Communication-Efficient Distributed Optimization

arXiv: https://arxiv.org/abs/1805.10111 · HTML: https://mlsanity.com/paper/1805.10111/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2018.

## Built on by (most cited first, 19 of 19)

- [1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed](https://mlsanity.com/paper/2102.02888/) (2021)
- [ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training](https://mlsanity.com/paper/2110.05323/) (2021)
- [Time-Correlated Sparsification for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2101.08837/) (2021)
- [1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed](https://mlsanity.com/paper/2104.06069/) (2021)
- [ResIST: Layer-Wise Decomposition of ResNets for Distributed Training](https://mlsanity.com/paper/2107.00961/) (2021)
- [Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression](https://mlsanity.com/paper/2301.09830/) (2023)
- [Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropout](https://mlsanity.com/paper/2210.16105/) (2022)
- [Low-Precision Training of Large Language Models: Methods, Challenges, and Opportunities](https://mlsanity.com/paper/2505.01043/) (2025)
- [Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Training](https://mlsanity.com/paper/2306.01381/) (2023)
- [An Expectation-Maximization Perspective on Federated Learning](https://mlsanity.com/paper/2111.10192/) (2021)
- [Distributed Adversarial Training to Robustify Deep Neural Networks at Scale](https://mlsanity.com/paper/2206.06257/) (2022)
- [Federated Learning over Noisy Channels: Convergence Analysis and Design Examples](https://mlsanity.com/paper/2101.02198/) (2021)
- [Efficient Direct-Connect Topologies for Collective Communications](https://mlsanity.com/paper/2202.03356/) (2022)
- [Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs](https://mlsanity.com/paper/2502.17009/) (2025)
- [Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise](https://mlsanity.com/paper/2307.07406/) (2023)
- [QC-ODKLA: Quantized and Communication-Censored Online Decentralized Kernel Learning via Linearized ADMM](https://mlsanity.com/paper/2208.02777/) (2022)
- [DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning](https://mlsanity.com/paper/2102.03112/) (2021)
- [Energy-Efficient Wireless Federated Learning via Doubly Adaptive Quantization](https://mlsanity.com/paper/2402.12957/) (2024)
- [Quantized Stochastic Primal-Dual Methods for Distributed Optimization under Relaxed Global Geometry](https://mlsanity.com/paper/2606.11339/) (2026)

## Abstract (arXiv)

Modern distributed training of machine learning models suffers from high communication overhead for synchronizing stochastic gradients and model parameters. In this paper, to reduce the communication complexity, we propose double quantization, a general scheme for quantizing both model parameters and gradients. Three communication-efficient algorithms are proposed under this general scheme. Specifically, (i) we propose a low-precision algorithm AsyLPG with asynchronous parallelism, (ii) we explore integrating gradient sparsification with double quantization and develop Sparse-AsyLPG, (iii) we show that double quantization can also be accelerated by momentum technique and design accelerated AsyLPG. We establish rigorous performance guarantees for the algorithms, and conduct experiments on a multi-server test-bed to demonstrate that our algorithms can effectively save transmitted bits without performance degradation.