# Gossip training for deep learning

arXiv: https://arxiv.org/abs/1611.09726 · HTML: https://mlsanity.com/paper/1611.09726/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2016.

## Built on by (most cited first, 30 of 35)

- [DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training](https://mlsanity.com/paper/2206.00187/) (2022)
- [Decentralized federated learning of deep neural networks on non-iid data](https://mlsanity.com/paper/2107.08517/) (2021)
- [An Energy and Carbon Footprint Analysis of Distributed and Federated Learning](https://mlsanity.com/paper/2206.10380/) (2022)
- [Decentralized Directed Collaboration for Personalized Federated Learning](https://mlsanity.com/paper/2405.17876/) (2024)
- [A Framework for Energy and Carbon Footprint Analysis of Distributed and Federated Edge Learning](https://mlsanity.com/paper/2103.10346/) (2021)
- [Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training](https://mlsanity.com/paper/2305.15157/) (2023)
- [Decentralized adaptive clustering of deep nets is beneficial for client collaboration](https://mlsanity.com/paper/2206.08839/) (2022)
- [A2CiD2: Accelerating Asynchronous Communication in Decentralized Deep Learning](https://mlsanity.com/paper/2306.08289/) (2023)
- [Network Gradient Descent Algorithm for Decentralized Federated Learning](https://mlsanity.com/paper/2205.08364/) (2022)
- [Olive Branch Learning: A Topology-Aware Federated Learning Framework for Space-Air-Ground Integrated Network](https://mlsanity.com/paper/2212.01215/) (2022)
- [DRACO: Decentralized Asynchronous Federated Learning over Row-Stochastic Wireless Networks](https://mlsanity.com/paper/2406.13533/) (2024)
- [OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters](https://mlsanity.com/paper/2503.17469/) (2025)
- [NoLoCo: No-all-reduce Low Communication Training Method for Large Models](https://mlsanity.com/paper/2506.10911/) (2025)
- [Efficient Node Selection in Private Personalized Decentralized Learning](https://mlsanity.com/paper/2301.12755/) (2023)
- [NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training](https://mlsanity.com/paper/2603.03597/) (2026)
- [Tram-FL: Routing-based Model Training for Decentralized Federated Learning](https://mlsanity.com/paper/2308.04762/) (2023)
- [Protocol Learning, Decentralized Frontier Risk and the No-Off Problem](https://mlsanity.com/paper/2412.07890/) (2024)
- [DeceFL: A Principled Decentralized Federated Learning Framework](https://mlsanity.com/paper/2107.07171/) (2021)
- [Asynchronous Training Schemes in Distributed Learning with Time Delay](https://mlsanity.com/paper/2208.13154/) (2022)
- [Accelerating Parallel Stochastic Gradient Descent via Non-blocking Mini-batches](https://mlsanity.com/paper/2211.00889/) (2022)
- [Factored Gossip DiLoCo: Reducing Blocking Communication in DiLoCo](https://mlsanity.com/paper/2606.22768/) (2026)
- [EventGraD: Event-Triggered Communication in Parallel Machine Learning](https://mlsanity.com/paper/2103.07454/) (2021)
- [RCD-SGD: Resource-Constrained Distributed SGD in Heterogeneous Environment via Submodular Partitioning](https://mlsanity.com/paper/2211.00839/) (2022)
- [Structured Cooperative Learning with Graphical Model Priors](https://mlsanity.com/paper/2306.09595/) (2023)
- [Collaborative Distributed Machine Learning](https://mlsanity.com/paper/2309.16584/) (2023)
- [Adjacent Leader Decentralized Stochastic Gradient Descent](https://mlsanity.com/paper/2405.11389/) (2024)
- [On the effects of similarity metrics in decentralized deep learning under distributional shift](https://mlsanity.com/paper/2409.10720/) (2024)
- [Load-Aware Training Scheduling for Model Circulation-based Decentralized Federated Learning](https://mlsanity.com/paper/2506.09769/) (2025)
- [Concept-aware clustering for decentralized deep learning under temporal shift](https://mlsanity.com/paper/2306.12768/) (2023)
- [Technical Report: On the Convergence of Gossip Learning in the Presence of Node Inaccessibility](https://mlsanity.com/paper/2401.09498/) (2024)

## Abstract (arXiv)

We address the issue of speeding up the training of convolutional networks. Here we study a distributed method adapted to stochastic gradient descent (SGD). The parallel optimization setup uses several threads, each applying individual gradient descents on a local variable. We propose a new way to share information between different threads inspired by gossip algorithms and showing good consensus convergence properties. Our method called GoSGD has the advantage to be fully asynchronous and decentralized. We compared our method to the recent EASGD in citeelastic on CIFAR-10 show encouraging results.