# FLECS-CGD: A Federated Learning Second-Order Framework via Compression and Sketching with Compressed Gradient Differences

arXiv: https://arxiv.org/abs/2210.09626 · HTML: https://mlsanity.com/paper/2210.09626/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 18 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Federated Optimization: Distributed Machine Learning for On-Device Intelligence](https://mlsanity.com/paper/1610.02527/) (2016)
- [SARAH: A Novel Method for Machine Learning Problems Using Stochastic Recursive Gradient](https://mlsanity.com/paper/1703.00102/) (2017)
- [Distributed Learning with Compressed Gradient Differences](https://mlsanity.com/paper/1901.09269/) (2019)
- [Natural Compression for Distributed Deep Learning](https://mlsanity.com/paper/1905.10988/) (2019)
- [Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization](https://mlsanity.com/paper/2002.11364/) (2020)
- [CoCoA: A General Framework for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1611.02189/) (2016)
- [Stochastic Distributed Learning with Gradient Quantization and Variance Reduction](https://mlsanity.com/paper/1904.05115/) (2019)
- [FedNL: Making Newton-Type Methods Applicable to Federated Learning](https://mlsanity.com/paper/2106.02969/) (2021)
- [Communication-Efficient Distributed Dual Coordinate Ascent](https://mlsanity.com/paper/1409.1458/) (2014)
- [Fast Convergence Rates for Distributed Non-Bayesian Learning](https://mlsanity.com/paper/1508.05161/) (2015)
- [Distributed Coordinate Descent Method for Learning with Big Data](https://mlsanity.com/paper/1310.2059/) (2013)
- [Distributed Optimization with Arbitrary Local Solvers](https://mlsanity.com/paper/1512.04039/) (2015)
- [Decentralize and Randomize: Faster Algorithm for Wasserstein Barycenters](https://mlsanity.com/paper/1806.03915/) (2018)
- [AI-SARAH: Adaptive and Implicit Stochastic Recursive Gradient Methods](https://mlsanity.com/paper/2102.09700/) (2021)
- [A Newton-Based Method for Nonconvex Optimization with Fast Evasion of Saddle Points](https://mlsanity.com/paper/1707.08028/) (2017)
- [Inexact SARAH Algorithm for Stochastic Optimization](https://mlsanity.com/paper/1811.10105/) (2018)
- [Distributed Learning With Sparsified Gradient Differences](https://mlsanity.com/paper/2202.02491/) (2022)
- [Multi-Agent Image Classification via Reinforcement Learning](https://mlsanity.com/paper/1905.04835/) (2019)
- [Communication-Efficient Distributed Optimization of Self-Concordant Empirical Loss](https://mlsanity.com/paper/1501.00263/) (2015)
- [Newton Method over Networks is Fast up to the Statistical Precision](https://mlsanity.com/paper/2102.06780/) (2021)
- [An Accelerated Second-Order Method for Distributed Stochastic Optimization](https://mlsanity.com/paper/2103.14392/) (2021)
- [A Stochastic Newton Algorithm for Distributed Convex Optimization](https://mlsanity.com/paper/2110.02954/) (2021)
- [Distributed Computation of Wasserstein Barycenters over Networks](https://mlsanity.com/paper/1803.02933/) (2018)
- [Hyperfast Second-Order Local Solvers for Efficient Statistically Preconditioned Distributed Optimization](https://mlsanity.com/paper/2102.08246/) (2021)
- [Distributed Block Coordinate Descent for Minimizing Partially Separable Functions](https://mlsanity.com/paper/1406.0238/) (2014)

## Built on by (most cited first, 2 of 2)

- [Advancements in Federated Learning: Models, Methods, and Privacy](https://mlsanity.com/paper/2302.11466/) (2023)
- [Simple Stepsize for Quasi-Newton Methods with Global Convergence Guarantees](https://mlsanity.com/paper/2508.19712/) (2025)

## Abstract (arXiv)

In the recent paper FLECS (Agafonov et al, FLECS: A Federated Learning Second-Order Framework via Compression and Sketching), the second-order framework FLECS was proposed for the Federated Learning problem. This method utilize compression of sketched Hessians to make communication costs low. However, the main bottleneck of FLECS is gradient communication without compression. In this paper, we propose the modification of FLECS with compressed gradient differences, which we call FLECS-CGD (FLECS with Compressed Gradient Differences) and make it applicable for stochastic optimization. Convergence guarantees are provided in strongly convex and nonconvex cases. Experiments show the practical benefit of proposed approach.