# Fully Decoupled Neural Network Learning Using Delayed Gradients

arXiv: https://arxiv.org/abs/1906.09108 · HTML: https://mlsanity.com/paper/1906.09108/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2019.

## Built on by (most cited first, 12 of 12)

- [PETRA: Parallel End-to-end Training with Reversible Architectures](https://mlsanity.com/paper/2406.02052/) (2024)
- [ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training](https://mlsanity.com/paper/2406.02613/) (2024)
- [Cortico-cerebellar networks as decoupling neural interfaces](https://mlsanity.com/paper/2110.11501/) (2021)
- [Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learning](https://mlsanity.com/paper/2207.11019/) (2022)
- [Asynchronous Training Schemes in Distributed Learning with Time Delay](https://mlsanity.com/paper/2208.13154/) (2022)
- [Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotation](https://mlsanity.com/paper/2602.03515/) (2026)
- [ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction](https://mlsanity.com/paper/2305.13236/) (2023)
- [Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework](https://mlsanity.com/paper/2105.09837/) (2021)
- [A Survey From Distributed Machine Learning to Distributed Deep Learning](https://mlsanity.com/paper/2307.05232/) (2023)
- [Frozen Backpropagation: Relaxing Weight Symmetry in Deep Spiking Neural Networks](https://mlsanity.com/paper/2505.13741/) (2025)
- [Hierarchical Feature-level Reverse Propagation for Post-Training Neural Networks](https://mlsanity.com/paper/2506.07188/) (2025)
- [Distributed Deep Learning using Stochastic Gradient Staleness](https://mlsanity.com/paper/2509.05679/) (2025)

## Abstract (arXiv)

Training neural networks with back-propagation (BP) requires a sequential passing of activations and gradients, which forces the network modules to work in a synchronous fashion. This has been recognized as the lockings (i.e., the forward, backward and update lockings) inherited from the BP. In this paper, we propose a fully decoupled training scheme using delayed gradients (FDG) to break all these lockings. The FDG splits a neural network into multiple modules and trains them independently and asynchronously using different workers (e.g., GPUs). We also introduce a gradient shrinking process to reduce the stale gradient effect caused by the delayed gradients. In addition, we prove that the proposed FDG algorithm guarantees a statistical convergence during training. Experiments are conducted by training deep convolutional neural networks to perform classification tasks on benchmark datasets, showing comparable or better results against the state-of-the-art methods as well as the BP in terms of both generalization and acceleration abilities. In particular, we show that the FDG is also able to train very wide networks (e.g., WRN-28-10) and extremely deep networks (e.g., ResNet-1202). Code is available at https://github.com/ZHUANGHP/FDG.