# GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent

arXiv: https://arxiv.org/abs/1803.05880 · HTML: https://mlsanity.com/paper/1803.05880/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2018.

## Built on by (most cited first, 19 of 19)

- [Opportunities of Federated Learning in Connected, Cooperative and Automated Industrial Systems](https://mlsanity.com/paper/2101.03367/) (2021)
- [Autonomy and Intelligence in the Computing Continuum: Challenges, Enablers, and Future Directions for Orchestration](https://mlsanity.com/paper/2205.01423/) (2022)
- [Backdoor Attacks in Peer-to-Peer Federated Learning](https://mlsanity.com/paper/2301.09732/) (2023)
- [A2CiD2: Accelerating Asynchronous Communication in Decentralized Deep Learning](https://mlsanity.com/paper/2306.08289/) (2023)
- [NoLoCo: No-all-reduce Low Communication Training Method for Large Models](https://mlsanity.com/paper/2506.10911/) (2025)
- [DACFL: Dynamic Average Consensus Based Federated Learning in Decentralized Topology](https://mlsanity.com/paper/2111.05505/) (2021)
- [Decentralized Graph Neural Network for Privacy-Preserving Recommendation](https://mlsanity.com/paper/2308.08072/) (2023)
- [DeceFL: A Principled Decentralized Federated Learning Framework](https://mlsanity.com/paper/2107.07171/) (2021)
- [p2pGNN: A Decentralized Graph Neural Network for Node Classification in Peer-to-Peer Networks](https://mlsanity.com/paper/2111.14837/) (2021)
- [Scatterbrained: A flexible and expandable pattern for decentralized machine learning](https://mlsanity.com/paper/2112.07718/) (2021)
- [EventGraD: Event-Triggered Communication in Parallel Machine Learning](https://mlsanity.com/paper/2103.07454/) (2021)
- [Runtime-Orchestrated Second-Order Optimization for Scalable LLM Training](https://mlsanity.com/paper/2605.16184/) (2026)
- [Detection of Insider Attacks in Distributed Projected Subgradient Algorithms](https://mlsanity.com/paper/2101.06917/) (2021)
- [Secure Distributed/Federated Learning: Prediction-Privacy Trade-Off for Multi-Agent System](https://mlsanity.com/paper/2205.04855/) (2022)
- [FLAGS Framework for Comparative Analysis of Federated Learning Algorithms](https://mlsanity.com/paper/2212.07179/) (2022)
- [On the Limit Performance of Floating Gossip](https://mlsanity.com/paper/2302.08413/) (2023)
- [Technical Report: On the Convergence of Gossip Learning in the Presence of Node Inaccessibility](https://mlsanity.com/paper/2401.09498/) (2024)
- [On Gossip Algorithms for Machine Learning with Pairwise Objectives](https://mlsanity.com/paper/2603.24128/) (2026)
- [Decentralized Gossip Learning and Federated Averaging for Histopathology Image Classification](https://mlsanity.com/paper/2609.16448/) (2026)

## Abstract (arXiv)

In this paper, we present GossipGraD - a gossip communication protocol based Stochastic Gradient Descent (SGD) algorithm for scaling Deep Learning (DL) algorithms on large-scale systems. The salient features of GossipGraD are: 1) reduction in overall communication complexity from Theta(log(p)) for p compute nodes in well-studied SGD to O(1), 2) model diffusion such that compute nodes exchange their updates (gradients) indirectly after every log(p) steps, 3) rotation of communication partners for facilitating direct diffusion of gradients, 4) asynchronous distributed shuffle of samples during the feedforward phase in SGD to prevent over-fitting, 5) asynchronous communication of gradients for further reducing the communication cost of SGD and GossipGraD. We implement GossipGraD for GPU and CPU clusters and use NVIDIA GPUs (Pascal P100) connected with InfiniBand, and Intel Knights Landing (KNL) connected with Aries network. We evaluate GossipGraD using well-studied dataset ImageNet-1K (~250GB), and widely studied neural network topologies such as GoogLeNet and ResNet50 (current winner of ImageNet Large Scale Visualization Research Challenge (ILSVRC)). Our performance evaluation using both KNL and Pascal GPUs indicates that GossipGraD can achieve perfect efficiency for these datasets and their associated neural network topologies. Specifically, for ResNet50, GossipGraD is able to achieve ~100% compute efficiency using 128 NVIDIA Pascal P100 GPUs - while matching the top-1 classification accuracy published in literature.