# Asynchronous Parallel Incremental Block-Coordinate Descent for Decentralized Machine Learning

arXiv: https://arxiv.org/abs/2202.03263 · HTML: https://mlsanity.com/paper/2202.03263/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 18 of 18)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [The Non-IID Data Quagmire of Decentralized Machine Learning](https://mlsanity.com/paper/1910.00189/) (2019)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [EXTRA: An Exact First-Order Algorithm for Decentralized Consensus Optimization](https://mlsanity.com/paper/1404.6264/) (2014)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [On Markov Chain Gradient Descent](https://mlsanity.com/paper/1809.04216/) (2018)
- [On the Linear Convergence of the ADMM in Decentralized Consensus Optimization](https://mlsanity.com/paper/1307.5561/) (2013)
- [Walkman: A Communication-Efficient Random-Walk Algorithm for Decentralized Optimization](https://mlsanity.com/paper/1804.06568/) (2018)
- [Asynchronous Stochastic Coordinate Descent: Parallelism and Convergence Properties](https://mlsanity.com/paper/1403.3862/) (2014)
- [Federated Learning over Wireless IoT Networks with Optimized Communication and Resources](https://mlsanity.com/paper/2110.11775/) (2021)
- [Distributed Nesterov gradient methods over arbitrary graphs](https://mlsanity.com/paper/1901.06995/) (2019)
- [Decentralized Consensus Optimization Based on Parallel Random Walk](https://mlsanity.com/paper/1911.09747/) (2019)
- [Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing](https://mlsanity.com/paper/2010.00914/) (2020)
- [Newton-like method with diagonal correction for distributed optimization](https://mlsanity.com/paper/1509.01703/) (2015)
- [On the Convergence of Asynchronous Parallel Iteration with Unbounded Delays](https://mlsanity.com/paper/1612.04425/) (2016)
- [Privacy-preserving Incremental ADMM for Decentralized Consensus Optimization](https://mlsanity.com/paper/2003.10615/) (2020)

## Built on by (most cited first, 1 of 1)

- [A Multi-Token Coordinate Descent Method for Semi-Decentralized Vertical Federated Learning](https://mlsanity.com/paper/2309.09977/) (2023)

## Abstract (arXiv)

Machine learning (ML) is a key technique for big-data-driven modelling and analysis of massive Internet of Things (IoT) based intelligent and ubiquitous computing. For fast-increasing applications and data amounts, distributed learning is a promising emerging paradigm since it is often impractical or inefficient to share/aggregate data to a centralized location from distinct ones. This paper studies the problem of training an ML model over decentralized systems, where data are distributed over many user devices and the learning algorithm run on-device, with the aim of relaxing the burden at a central entity/server. Although gossip-based approaches have been used for this purpose in different use cases, they suffer from high communication costs, especially when the number of devices is large. To mitigate this, incremental-based methods are proposed. We first introduce incremental block-coordinate descent (I-BCD) for the decentralized ML, which can reduce communication costs at the expense of running time. To accelerate the convergence speed, an asynchronous parallel incremental BCD (API-BCD) method is proposed, where multiple devices/agents are active in an asynchronous fashion. We derive convergence properties for the proposed methods. Simulation results also show that our API-BCD method outperforms state of the art in terms of running time and communication costs.