# Communication-Efficient {Federated} Learning Using Censored Heavy Ball Descent

arXiv: https://arxiv.org/abs/2209.11944 · HTML: https://mlsanity.com/paper/2209.11944/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 24 Sep 2022.

## Builds on (most cited first, 29 of 29)

- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Sparsified SGD with Memory](https://mlsanity.com/paper/1809.07599/) (2018)
- [Communication Efficient Distributed Optimization using an Approximate Newton-type Method](https://mlsanity.com/paper/1312.7853/) (2013)
- [Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients](https://mlsanity.com/paper/2102.07053/) (2021)
- [LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning](https://mlsanity.com/paper/1805.09965/) (2018)
- [A Stochastic Quasi-Newton Method for Large-Scale Optimization](https://mlsanity.com/paper/1401.7020/) (2014)
- [CoCoA: A General Framework for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1611.02189/) (2016)
- [Adaptive Quantization of Model Updates for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2102.04487/) (2021)
- [Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applications](https://mlsanity.com/paper/2008.02608/) (2020)
- [Global convergence of the Heavy-ball method for convex optimization](https://mlsanity.com/paper/1412.7457/) (2014)
- [SparseFed: Mitigating Model Poisoning Attacks in Federated Learning with Sparsification](https://mlsanity.com/paper/2112.06274/) (2021)
- [AIDE: Fast and Communication Efficient Distributed Optimization](https://mlsanity.com/paper/1608.06879/) (2016)
- [Optimal Algorithms for Non-Smooth Distributed Optimization in Networks](https://mlsanity.com/paper/1806.00291/) (2018)
- [DAdaQuant: Doubly-adaptive quantization for communication-efficient Federated Learning](https://mlsanity.com/paper/2111.00465/) (2021)
- [A Linearly-Convergent Stochastic L-BFGS Algorithm](https://mlsanity.com/paper/1508.02087/) (2015)
- [Quantized Federated Learning under Transmission Delay and Outage Constraints](https://mlsanity.com/paper/2106.09397/) (2021)
- [Communication-Efficient Federated Learning with Compensated Overlap-FedAvg](https://mlsanity.com/paper/2012.06706/) (2020)
- [COLA: Decentralized Linear Learning](https://mlsanity.com/paper/1808.04883/) (2018)
- [Approximate Byzantine Fault-Tolerance in Distributed Optimization](https://mlsanity.com/paper/2101.09337/) (2021)
- [Time-Correlated Sparsification for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2101.08837/) (2021)
- [Stochastic Block BFGS: Squeezing More Curvature out of Data](https://mlsanity.com/paper/1603.09649/) (2016)
- [Distributed Learning in Non-Convex Environments -- Part I: Agreement at a Linear Rate](https://mlsanity.com/paper/1907.01848/) (2019)
- [Secure Aggregation with Heterogeneous Quantization in Federated Learning](https://mlsanity.com/paper/2009.14388/) (2020)
- [Distributed Learning With Sparsified Gradient Differences](https://mlsanity.com/paper/2202.02491/) (2022)
- [Robust Distributed Optimization With Randomly Corrupted Gradients](https://mlsanity.com/paper/2106.14956/) (2021)
- [Communication Efficient Federated Learning via Ordered ADMM in a Fully Decentralized Setting](https://mlsanity.com/paper/2202.02580/) (2022)
- [Distributed Aggregative Optimization over Multi-Agent Networks](https://mlsanity.com/paper/2005.13436/) (2020)
- [Ordering for Communication-Efficient Quickest Change Detection in a Decomposable Graphical Model](https://mlsanity.com/paper/2008.03871/) (2020)
- [Testing the Structure of a Gaussian Graphical Model with Reduced Transmissions in a Distributed Setting](https://mlsanity.com/paper/1905.01376/) (2019)

## Built on by (most cited first, 1 of 1)

- [FedUHB: Accelerating Federated Unlearning via Polyak Heavy Ball Method](https://mlsanity.com/paper/2411.11039/) (2024)

## Abstract (arXiv)

Distributed machine learning enables scalability and computational offloading, but requires significant levels of communication. Consequently, communication efficiency in distributed learning settings is an important consideration, especially when the communications are wireless and battery-driven devices are employed. In this paper we develop a censoring-based heavy ball (CHB) method for distributed learning in a server-worker architecture. Each worker self-censors unless its local gradient is sufficiently different from the previously transmitted one. The significant practical advantages of the HB method for learning problems are well known, but the question of reducing communications has not been addressed. CHB takes advantage of the HB smoothing to eliminate reporting small changes, and provably achieves a linear convergence rate equivalent to that of the classical HB method for smooth and strongly convex objective functions. The convergence guarantee of CHB is theoretically justified for both convex and nonconvex cases. In addition we prove that, under some conditions, at least half of all communications can be eliminated without any impact on convergence rate. Extensive numerical results validate the communication efficiency of CHB on both synthetic and real datasets, for convex, nonconvex, and nondifferentiable cases. Given a target accuracy, CHB can significantly reduce the number of communications compared to existing algorithms, achieving the same accuracy without slowing down the optimization process.