# Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients

arXiv: https://arxiv.org/abs/1909.07588 · HTML: https://mlsanity.com/paper/1909.07588/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2019.

## Built on by (most cited first, 30 of 32)

- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [Decentralized Federated Averaging](https://mlsanity.com/paper/2104.11375/) (2021)
- [1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed](https://mlsanity.com/paper/2102.02888/) (2021)
- [Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2102.04487/) (2021)
- [3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation](https://mlsanity.com/paper/2202.00998/) (2022)
- [1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed](https://mlsanity.com/paper/2104.06069/) (2021)
- [FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learning](https://mlsanity.com/paper/2206.08829/) (2022)
- [GradSkip: Communication-Accelerated Local Gradient Methods with Better Computational Complexity](https://mlsanity.com/paper/2210.16402/) (2022)
- [Adaptive Compression for Communication-Efficient Distributed Training](https://mlsanity.com/paper/2211.00188/) (2022)
- [Distributed Distributionally Robust Optimization with Non-Convex Objectives](https://mlsanity.com/paper/2210.07588/) (2022)
- [BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning](https://mlsanity.com/paper/2104.06685/) (2021)
- [Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation](https://mlsanity.com/paper/2206.03588/) (2022)
- [DeMo: Decoupled Momentum Optimization](https://mlsanity.com/paper/2411.19870/) (2024)
- [Distributed Linear Bandits under Communication Constraints](https://mlsanity.com/paper/2211.02212/) (2022)
- [Leveraging Spatial and Temporal Correlations in Sparsified Mean Estimation](https://mlsanity.com/paper/2110.07751/) (2021)
- [A Communication-Efficient Adaptive Algorithm for Federated Learning under Cumulative Regret](https://mlsanity.com/paper/2301.08869/) (2023)
- [Personalized Federated Learning with Server-Side Information](https://mlsanity.com/paper/2205.11044/) (2022)
- [Blockchain-Based Federated Learning: Incentivizing Data Sharing and Penalizing Dishonest Behavior](https://mlsanity.com/paper/2307.10492/) (2023)
- [FedAQ: Communication-Efficient Federated Edge Learning via Joint Uplink and Downlink Adaptive Quantization](https://mlsanity.com/paper/2406.18156/) (2024)
- [FedCau: A Proactive Stop Policy for Communication and Computation Efficient Federated Learning](https://mlsanity.com/paper/2204.07773/) (2022)
- [Federated Distributionally Robust Optimization with Non-Convex Objectives: Algorithm and Analysis](https://mlsanity.com/paper/2307.14364/) (2023)
- [Distributed Learning based on 1-Bit Gradient Coding in the Presence of Stragglers](https://mlsanity.com/paper/2403.14716/) (2024)
- [Federated Learning Using Three-Operator ADMM](https://mlsanity.com/paper/2211.04152/) (2022)
- [Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Scheme](https://mlsanity.com/paper/2404.15943/) (2024)
- [Lazy Queries Can Reduce Variance in Zeroth-order Optimization](https://mlsanity.com/paper/2206.07126/) (2022)
- [FedDIP: Federated Learning with Extreme Dynamic Pruning and Incremental Regularization](https://mlsanity.com/paper/2309.06805/) (2023)
- [Socialized Learning: A Survey of the Paradigm Shift for Edge Intelligence in Networked Systems](https://mlsanity.com/paper/2404.13348/) (2024)
- [SparDL: Distributed Deep Learning Training with Efficient Sparse Communication](https://mlsanity.com/paper/2304.00737/) (2023)
- [Communication-Efficient Zeroth-Order Distributed Online Optimization: Algorithm, Theory, and Applications](https://mlsanity.com/paper/2306.05655/) (2023)
- [Communication-Efficient Federated Learning via Regularized Sparse Random Networks](https://mlsanity.com/paper/2309.10834/) (2023)

## Abstract (arXiv)

The present paper develops a novel aggregated gradient approach for distributed machine learning that adaptively compresses the gradient communication. The key idea is to first quantize the computed gradients, and then skip less informative quantized gradient communications by reusing outdated gradients. Quantizing and skipping result in `lazy' worker-server communications, which justifies the term Lazily Aggregated Quantized gradient that is henceforth abbreviated as LAQ. Our LAQ can provably attain the same linear convergence rate as the gradient descent in the strongly convex case, while effecting major savings in the communication overhead both in transmitted bits as well as in communication rounds. Empirically, experiments with real data corroborate a significant communication reduction compared to existing gradient- and stochastic gradient-based algorithms.