# Quantization for decentralized learning under subspace constraints

arXiv: https://arxiv.org/abs/2209.07821 · HTML: https://mlsanity.com/paper/2209.07821/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 46 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 24 of 24)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Multi-Task Learning](https://mlsanity.com/paper/1705.10467/) (2017)
- [Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication](https://mlsanity.com/paper/1902.00340/) (2019)
- [QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding](https://mlsanity.com/paper/1610.02132/) (2016)
- [SGD and Hogwild! Convergence Without the Bounded Gradients Assumption](https://mlsanity.com/paper/1802.03801/) (2018)
- [Communication Compression for Decentralized Training](https://mlsanity.com/paper/1803.06443/) (2018)
- [A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!](https://mlsanity.com/paper/2011.01697/) (2020)
- [An Exact Quantized Decentralized Gradient Descent Algorithm](https://mlsanity.com/paper/1806.11536/) (2018)
- [SQuARM-SGD: Communication-Efficient Momentum SGD for Decentralized Optimization](https://mlsanity.com/paper/2005.07041/) (2020)
- [Multitask learning over graphs: An Approach for Distributed, Streaming Machine Learning](https://mlsanity.com/paper/2001.02112/) (2020)
- [Quantized Decentralized Stochastic Learning over Directed Graphs](https://mlsanity.com/paper/2002.09964/) (2020)
- [Finite-Bit Quantization For Distributed Algorithms With Linear Convergence](https://mlsanity.com/paper/2107.11304/) (2021)
- [Learning over Multitask Graphs -- Part I: Stability Analysis](https://mlsanity.com/paper/1805.08535/) (2018)
- [Multitask Diffusion Adaptation over Networks](https://mlsanity.com/paper/1311.4894/) (2013)
- [Adaptation and learning over networks under subspace constraints -- Part I: Stability Analysis](https://mlsanity.com/paper/1905.08750/) (2019)
- [Distributed Optimization With Local Domains: Applications in MPC and Network Flows](https://mlsanity.com/paper/1305.1885/) (2013)
- [Distributed Coupled Multi-Agent Stochastic Optimization](https://mlsanity.com/paper/1712.08817/) (2017)
- [Adaptation and learning over networks under subspace constraints -- Part II: Performance Analysis](https://mlsanity.com/paper/1906.12250/) (2019)
- [Compressed Distributed Gradient Descent: Communication-Efficient Consensus over Networks](https://mlsanity.com/paper/1812.04048/) (2018)
- [Distributed Adaptive Learning Under Communication Constraints](https://mlsanity.com/paper/2112.02129/) (2021)
- [Diffusion LMS for Multitask Problems with Local Linear Equality Constraints](https://mlsanity.com/paper/1610.02943/) (2016)
- [CIRFE: A Distributed Random Fields Estimator](https://mlsanity.com/paper/1802.04943/) (2018)

## Built on by (most cited first, 7 of 7)

- [Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions](https://mlsanity.com/paper/2501.04437/) (2025)
- [Communication-efficient Vertical Federated Learning via Compressed Error Feedback](https://mlsanity.com/paper/2406.14420/) (2024)
- [Differential error feedback for communication-efficient decentralized learning](https://mlsanity.com/paper/2406.18418/) (2024)
- [Compressed Regression over Adaptive Networks](https://mlsanity.com/paper/2304.03638/) (2023)
- [Distributed Training of Large Graph Neural Networks with Variable Communication Rates](https://mlsanity.com/paper/2406.17611/) (2024)
- [Quantitative Error Feedback for Quantization Noise Reduction of Filtering over Graphs](https://mlsanity.com/paper/2506.01404/) (2025)
- [Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training](https://mlsanity.com/paper/2606.16384/) (2026)

## Abstract (arXiv)

In this paper, we consider decentralized optimization problems where agents have individual cost functions to minimize subject to subspace constraints that require the minimizers across the network to lie in low-dimensional subspaces. This constrained formulation includes consensus or single-task optimization as special cases, and allows for more general task relatedness models such as multitask smoothness and coupled optimization. In order to cope with communication constraints, we propose and study an adaptive decentralized strategy where the agents employ differential randomized quantizers to compress their estimates before communicating with their neighbors. The analysis shows that, under some general conditions on the quantization noise, and for sufficiently small step-sizes μ, the strategy is stable both in terms of mean-square error and average bit rate: by reducing μ, it is possible to keep the estimation errors small (on the order of μ) without increasing indefinitely the bit rate as μrightarrow 0. Simulations illustrate the theoretical findings and the effectiveness of the proposed approach, revealing that decentralized learning is achievable at the expense of only a few bits.