# Decentralized Deep Learning using Momentum-Accelerated Consensus

arXiv: https://arxiv.org/abs/2010.11166 · HTML: https://mlsanity.com/paper/2010.11166/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2020.

## Built on by (most cited first, 14 of 14)

- [Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data](https://mlsanity.com/paper/2102.04761/) (2021)
- [Exponential Graph is Provably Efficient for Decentralized Deep Training](https://mlsanity.com/paper/2110.13363/) (2021)
- [DecentLaM: Decentralized Momentum SGD for Large-batch Deep Training](https://mlsanity.com/paper/2104.11981/) (2021)
- [Cross-Gradient Aggregation for Decentralized Learning from Non-IID data](https://mlsanity.com/paper/2103.02051/) (2021)
- [Global Update Tracking: A Decentralized Learning Algorithm for Heterogeneous Data](https://mlsanity.com/paper/2305.04792/) (2023)
- [Distributed Learning over Networks with Graph-Attention-Based Personalization](https://mlsanity.com/paper/2305.13041/) (2023)
- [Neighborhood Gradient Clustering: An Efficient Decentralized Learning Method for Non-IID Data Distributions](https://mlsanity.com/paper/2209.14390/) (2022)
- [DIMAT: Decentralized Iterative Merging-And-Training for Deep Learning Models](https://mlsanity.com/paper/2404.08079/) (2024)
- [CoDeC: Communication-Efficient Decentralized Continual Learning](https://mlsanity.com/paper/2303.15378/) (2023)
- [Generalizing Differentially Private Decentralized Deep Learning with Multi-Agent Consensus](https://mlsanity.com/paper/2306.13892/) (2023)
- [ROSS: RObust decentralized Stochastic learning based on Shapley values](https://mlsanity.com/paper/2411.00365/) (2024)
- [Cross-feature Contrastive Loss for Decentralized Deep Learning on Heterogeneous Data](https://mlsanity.com/paper/2310.15890/) (2023)
- [SADDLe: Sharpness-Aware Decentralized Deep Learning with Heterogeneous Data](https://mlsanity.com/paper/2405.13961/) (2024)
- [Resilient Peer-to-peer Learning based on Adaptive Aggregation](https://mlsanity.com/paper/2501.04610/) (2025)

## Abstract (arXiv)

We consider the problem of decentralized deep learning where multiple agents collaborate to learn from a distributed dataset. While there exist several decentralized deep learning approaches, the majority consider a central parameter-server topology for aggregating the model parameters from the agents. However, such a topology may be inapplicable in networked systems such as ad-hoc mobile networks, field robotics, and power network systems where direct communication with the central parameter server may be inefficient. In this context, we propose and analyze a novel decentralized deep learning algorithm where the agents interact over a fixed communication topology (without a central server). Our algorithm is based on the heavy-ball acceleration method used in gradient-based optimization. We propose a novel consensus protocol where each agent shares with its neighbors its model parameters as well as gradient-momentum values during the optimization process. We consider both strongly convex and non-convex objective functions and theoretically analyze our algorithm's performance. We present several empirical comparisons with competing decentralized learning methods to demonstrate the efficacy of our approach under different communication topologies.