# Log-Scale Quantization in Distributed First-Order Methods: Gradient-based Learning from Distributed Data

arXiv: https://arxiv.org/abs/2406.00621 · HTML: https://mlsanity.com/paper/2406.00621/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 66 references that are in the map.
- First posted on arXiv 2 Jun 2024.

## Builds on (most cited first, 27 of 27)

- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization](https://mlsanity.com/paper/1709.08765/) (2017)
- [Optimal algorithms for smooth and strongly convex distributed optimization in networks](https://mlsanity.com/paper/1702.08704/) (2017)
- [Distributed optimization over time-varying directed graphs](https://mlsanity.com/paper/1303.2289/) (2013)
- [Exponential Graph is Provably Efficient for Decentralized Deep Training](https://mlsanity.com/paper/2110.13363/) (2021)
- [Stochastic Gradient-Push for Strongly Convex Functions on Time-Varying Directed Graphs](https://mlsanity.com/paper/1406.2075/) (2014)
- [Accelerated Distributed Nesterov Gradient Descent](https://mlsanity.com/paper/1705.07176/) (2017)
- [Robust Asynchronous Stochastic Gradient-Push: Asymptotically Optimal and Network-Independent Performance for Strongly Convex Functions](https://mlsanity.com/paper/1811.03982/) (2018)
- [Cloud K-SVD: A Collaborative Dictionary Learning Algorithm for Big, Distributed Data](https://mlsanity.com/paper/1412.7839/) (2014)
- [Convergence rates for distributed stochastic optimization over random networks](https://mlsanity.com/paper/1803.07836/) (2018)
- [ADD-OPT: Accelerated Distributed Directed Optimization](https://mlsanity.com/paper/1607.04757/) (2016)
- [A fast randomized incremental gradient method for decentralized non-convex optimization](https://mlsanity.com/paper/2011.03853/) (2020)
- [S-ADDOPT: Decentralized stochastic first-order optimization over directed graphs](https://mlsanity.com/paper/2005.07785/) (2020)
- [Compressed Gradient Tracking for Decentralized Optimization Over General Directed Networks](https://mlsanity.com/paper/2106.07243/) (2021)
- [Distributed support-vector-machine over dynamic balanced directed networks](https://mlsanity.com/paper/2104.00399/) (2021)
- [A Compressed Gradient Tracking Method for Decentralized Optimization with Linear Convergence](https://mlsanity.com/paper/2205.12623/) (2022)
- [Beyond Exponential Graph: Communication-Efficient Topologies for Decentralized Learning via Finite-time Convergence](https://mlsanity.com/paper/2305.11420/) (2023)
- [Quantized Distributed Gradient Tracking Algorithm with Linear Convergence in Directed Networks](https://mlsanity.com/paper/2104.03649/) (2021)
- [Push-SAGA: A decentralized stochastic algorithm with variance reduction over directed graphs](https://mlsanity.com/paper/2008.06082/) (2020)
- [Fast-Convergent Dynamics for Distributed Allocation of Resources Over Switching Sparse Networks with Quantized Communication Links](https://mlsanity.com/paper/2012.08181/) (2020)
- [Consensus-Based Distributed Estimation in the Presence of Heterogeneous, Time-Invariant Delays](https://mlsanity.com/paper/2104.00394/) (2021)
- [Accelerated AB/Push-Pull Methods for Distributed Optimization over Time-Varying Directed Networks](https://mlsanity.com/paper/2302.01214/) (2023)
- [Discretized Distributed Optimization over Dynamic Digraphs](https://mlsanity.com/paper/2311.07939/) (2023)
- [Quantization Design for Distributed Optimization](https://mlsanity.com/paper/1504.02317/) (2015)
- [Distributed Delay-Tolerant Strategies for Equality-Constraint Sum-Preserving Resource Allocation](https://mlsanity.com/paper/2310.18225/) (2023)
- [Geometric Convergence of Distributed Heavy-Ball Nash Equilibrium Algorithm over Time-Varying Digraphs with Unconstrained Actions](https://mlsanity.com/paper/2303.16385/) (2023)
- [Distributed CPU Scheduling Subject to Nonlinear Constraints](https://mlsanity.com/paper/2208.14059/) (2022)

## Built on by (most cited first, 4 of 4)

- [Communication-Efficient Personalized Distributed Learning with Data and Node Heterogeneity](https://mlsanity.com/paper/2504.17520/) (2025)
- [A Safety-Oriented Self-Learning Algorithm for Autonomous Driving: Evolution Starting from a Basic Model](https://mlsanity.com/paper/2408.12190/) (2024)
- [Distributed gradient methods under heavy-tailed communication noise](https://mlsanity.com/paper/2505.24464/) (2025)
- [Quantitative Error Feedback for Quantization Noise Reduction of Filtering over Graphs](https://mlsanity.com/paper/2506.01404/) (2025)

## Abstract (arXiv)

Decentralized strategies are of interest for learning from large-scale data over networks. This paper studies learning over a network of geographically distributed nodes/agents subject to quantization. Each node possesses a private local cost function, collectively contributing to a global cost function, which the considered methodology aims to minimize. In contrast to many existing papers, the information exchange among nodes is log-quantized to address limited network-bandwidth in practical situations. We consider a first-order computationally efficient distributed optimization algorithm (with no extra inner consensus loop) that leverages node-level gradient correction based on local data and network-level gradient aggregation only over nearby nodes. This method only requires balanced networks with no need for stochastic weight design. It can handle log-scale quantized data exchange over possibly time-varying and switching network setups. We study convergence over both structured networks (for example, training over data-centers) and ad-hoc multi-agent networks (for example, training over dynamic robotic networks). Through experimental validation, we show that (i) structured networks generally result in a smaller optimality gap, and (ii) log-scale quantization leads to a smaller optimality gap compared to uniform quantization.