# LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning

arXiv: https://arxiv.org/abs/1805.09965 · HTML: https://mlsanity.com/paper/1805.09965/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 May 2018.

## Built on by (most cited first, 30 of 96)

- [Distributed Learning in Wireless Networks: Recent Progress and Future Challenges](https://mlsanity.com/paper/2104.02151/) (2021)
- [Decentralized Federated Averaging](https://mlsanity.com/paper/2104.11375/) (2021)
- [CAFE: Catastrophic Data Leakage in Vertical Federated Learning](https://mlsanity.com/paper/2110.15122/) (2021)
- [Exponential Graph is Provably Efficient for Decentralized Deep Training](https://mlsanity.com/paper/2110.13363/) (2021)
- [Federated Learning via Inexact ADMM](https://mlsanity.com/paper/2204.10607/) (2022)
- [Anarchic Federated Learning](https://mlsanity.com/paper/2108.09875/) (2021)
- [DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training](https://mlsanity.com/paper/2104.11981/) (2021)
- [Dynamic Scheduling for Over-the-Air Federated Edge Learning with Energy Constraints](https://mlsanity.com/paper/2106.00490/) (2021)
- [Lower Bounds and Nearly Optimal Algorithms in Distributed Learning with Communication Compression](https://mlsanity.com/paper/2206.03665/) (2022)
- [Federated Learning Under Intermittent Client Availability and Time-Varying Communication Constraints](https://mlsanity.com/paper/2205.06730/) (2022)
- [3PC: Three Point Compressors for Communication-Efficient Distributed Training and a Better Theory for Lazy Aggregation](https://mlsanity.com/paper/2202.00998/) (2022)
- [Accelerating Gossip SGD with Periodic Global Averaging](https://mlsanity.com/paper/2105.09080/) (2021)
- [FedNew: A Communication-Efficient and Privacy-Preserving Newton-Type Method for Federated Learning](https://mlsanity.com/paper/2206.08829/) (2022)
- [AsyncFedED: Asynchronous Federated Learning with Euclidean Distance based Adaptive Weight Aggregation](https://mlsanity.com/paper/2205.13797/) (2022)
- [Asynchronous Iterations in Optimization: New Sequence Results and Sharper Algorithmic Guarantees](https://mlsanity.com/paper/2109.04522/) (2021)
- [BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning](https://mlsanity.com/paper/2111.04287/) (2021)
- [Adaptive Compression for Communication-Efficient Distributed Training](https://mlsanity.com/paper/2211.00188/) (2022)
- [Adaptive Transmission Scheduling in Wireless Networks for Asynchronous Federated Learning](https://mlsanity.com/paper/2103.01422/) (2021)
- [Joint Optimization of Communications and Federated Learning Over the Air](https://mlsanity.com/paper/2104.03490/) (2021)
- [Metadata Normalization](https://mlsanity.com/paper/2104.09052/) (2021)
- [Accelerated Gradient Descent Learning over Multiple Access Fading Channels](https://mlsanity.com/paper/2107.12452/) (2021)
- [Communication-Efficient ADMM-based Federated Learning](https://mlsanity.com/paper/2110.15318/) (2021)
- [Unbiased Compression Saves Communication in Distributed Optimization: When and How Much?](https://mlsanity.com/paper/2305.16297/) (2023)
- [BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning](https://mlsanity.com/paper/2104.06685/) (2021)
- [Federated Learning over Wireless IoT Networks with Optimized Communication and Resources](https://mlsanity.com/paper/2110.11775/) (2021)
- [1-Bit Compressive Sensing for Efficient Federated Learning Over the Air](https://mlsanity.com/paper/2103.16055/) (2021)
- [SHED: A Newton-type algorithm for federated learning based on incremental Hessian eigenvector sharing](https://mlsanity.com/paper/2202.05800/) (2022)
- [Distributed Newton-Type Methods with Communication Compression and Bernoulli Aggregation](https://mlsanity.com/paper/2206.03588/) (2022)
- [Lower Bounds and Accelerated Algorithms in Distributed Stochastic Optimization with Communication Compression](https://mlsanity.com/paper/2305.07612/) (2023)
- [Bandwidth Allocation for Multiple Federated Learning Services in Wireless Edge Networks](https://mlsanity.com/paper/2101.03627/) (2021)

## Abstract (arXiv)

This paper presents a new class of gradient methods for distributed machine learning that adaptively skip the gradient calculations to learn with reduced communication and computation. Simple rules are designed to detect slowly-varying gradients and, therefore, trigger the reuse of outdated gradients. The resultant gradient-based algorithms are termed Lazily Aggregated Gradient --- justifying our acronym LAG used henceforth. Theoretically, the merits of this contribution are: i) the convergence rate is the same as batch gradient descent in strongly-convex, convex, and nonconvex smooth cases; and, ii) if the distributed datasets are heterogeneous (quantified by certain measurable constants), the communication rounds needed to achieve a targeted accuracy are reduced thanks to the adaptive reuse of lagged gradients. Numerical experiments on both synthetic and real data corroborate a significant communication reduction compared to alternatives.