# EventGraD: Event-Triggered Communication in Parallel Machine Learning

arXiv: https://arxiv.org/abs/2103.07454 · HTML: https://mlsanity.com/paper/2103.07454/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 56 references that are in the map.
- First posted on arXiv 12 Mar 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent](https://mlsanity.com/paper/1705.09056/) (2017)
- [Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training](https://mlsanity.com/paper/1712.01887/) (2017)
- [Robust and Communication-Efficient Federated Learning from Non-IID Data](https://mlsanity.com/paper/1903.02891/) (2019)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [The Convergence of Sparsified Gradient Methods](https://mlsanity.com/paper/1809.10505/) (2018)
- [Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification, and Local Computations](https://mlsanity.com/paper/1906.02367/) (2019)
- [On the Convergence of Decentralized Gradient Descent](https://mlsanity.com/paper/1310.7063/) (2013)
- [Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/1802.09941/) (2018)
- [Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization](https://mlsanity.com/paper/1506.08272/) (2015)
- [Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation](https://mlsanity.com/paper/1903.07424/) (2019)
- [Deep learning with Elastic Averaging SGD](https://mlsanity.com/paper/1412.6651/) (2014)
- [Ternary Compression for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2003.03564/) (2020)
- [Gossip training for deep learning](https://mlsanity.com/paper/1611.09726/) (2016)
- [SparCML: High-Performance Sparse Communication for Machine Learning](https://mlsanity.com/paper/1802.08021/) (2018)
- [Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training](https://mlsanity.com/paper/1811.03619/) (2018)
- [GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent](https://mlsanity.com/paper/1803.05880/) (2018)
- [How to scale distributed deep learning?](https://mlsanity.com/paper/1611.04581/) (2016)
- [Event-Triggered Communication and Control of Networked Systems for Multi-Agent Consensus](https://mlsanity.com/paper/1712.00429/) (2017)
- [Distributed Deep Learning with Event-Triggered Communication](https://mlsanity.com/paper/1909.05020/) (2019)
- [Distributed linear programming with event-triggered communication](https://mlsanity.com/paper/1405.0535/) (2014)

## Built on by (most cited first, 1 of 1)

- [An Energy Optimized Specializing DAG Federated Learning based on Event Triggered Communication](https://mlsanity.com/paper/2209.12531/) (2022)

## Abstract (arXiv)

Communication in parallel systems imposes significant overhead which often turns out to be a bottleneck in parallel machine learning. To relieve some of this overhead, in this paper, we present EventGraD - an algorithm with event-triggered communication for stochastic gradient descent in parallel machine learning. The main idea of this algorithm is to modify the requirement of communication at every iteration in standard implementations of stochastic gradient descent in parallel machine learning to communicating only when necessary at certain iterations. We provide theoretical analysis of convergence of our proposed algorithm. We also implement the proposed algorithm for data-parallel training of a popular residual neural network used for training the CIFAR-10 dataset and show that EventGraD can reduce the communication load by up to 60% while retaining the same level of accuracy. In addition, EventGraD can be combined with other approaches such as Top-K sparsification to decrease communication further while maintaining accuracy.