# Understanding Synthetic Gradients and Decoupled Neural Interfaces

arXiv: https://arxiv.org/abs/1703.00522 · HTML: https://mlsanity.com/paper/1703.00522/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Mar 2017.

## Built on by (most cited first, 30 of 35)

- [On Training Implicit Models](https://mlsanity.com/paper/2111.05177/) (2021)
- [Hebbian Deep Learning Without Feedback](https://mlsanity.com/paper/2209.11883/) (2022)
- [SoftHebb: Bayesian Inference in Unsupervised Hebbian Soft Winner-Take-All Networks](https://mlsanity.com/paper/2107.05747/) (2021)
- [Signal Propagation: A Framework for Learning and Inference In a Forward Pass](https://mlsanity.com/paper/2204.01723/) (2022)
- [Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence](https://mlsanity.com/paper/2106.01288/) (2021)
- [Deep Incubation: Training Large Models by Divide-and-Conquering](https://mlsanity.com/paper/2212.04129/) (2022)
- [Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning](https://mlsanity.com/paper/2106.06401/) (2021)
- [Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients](https://mlsanity.com/paper/2410.17764/) (2024)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Gradient Forward-Propagation for Large-Scale Temporal Video Modelling](https://mlsanity.com/paper/2106.08318/) (2021)
- [Counter-Current Learning: A Biologically Plausible Dual Network Approach for Deep Learning](https://mlsanity.com/paper/2409.19841/) (2024)
- [Module-wise Training of Neural Networks via the Minimizing Movement Scheme](https://mlsanity.com/paper/2309.17357/) (2023)
- [Self-Selected Attention Span for Accelerating Large Language Model Inference](https://mlsanity.com/paper/2404.09336/) (2024)
- [Target Propagation via Regularized Inversion](https://mlsanity.com/paper/2112.01453/) (2021)
- [Feed-Forward Optimization With Delayed Feedback for Neural Network Training](https://mlsanity.com/paper/2304.13372/) (2023)
- [LightFF: Lightweight Inference for Forward-Forward Algorithm](https://mlsanity.com/paper/2404.05241/) (2024)
- [Adaptive Spatial Goodness Encoding: Advancing and Scaling Forward-Forward Learning Without Backpropagation](https://mlsanity.com/paper/2509.12394/) (2025)
- [A Taxonomy of Recurrent Learning Rules](https://mlsanity.com/paper/2207.11439/) (2022)
- [Gradient Prediction with Control Variates in the Cheap-Forward Regime](https://mlsanity.com/paper/2511.05187/) (2025)
- [Block-wise Training of Residual Networks via the Minimizing Movement Scheme](https://mlsanity.com/paper/2210.00949/) (2022)
- [ADA-GP: Accelerating DNN Training By Adaptive Gradient Prediction](https://mlsanity.com/paper/2305.13236/) (2023)
- [Learning Using a Single Forward Pass](https://mlsanity.com/paper/2402.09769/) (2024)
- [Forward Learning of Graph Neural Networks](https://mlsanity.com/paper/2403.11004/) (2024)
- [Traces Propagation: Memory-Efficient and Scalable Forward-Only Learning in Spiking Neural Networks](https://mlsanity.com/paper/2509.13053/) (2025)
- [Fast Adversarial Attacks with Gradient Prediction](https://mlsanity.com/paper/2605.14868/) (2026)
- [Perceptual Gradient Networks](https://mlsanity.com/paper/2105.01957/) (2021)
- [Front Contribution instead of Back Propagation](https://mlsanity.com/paper/2106.05569/) (2021)
- [Learning Multi-Layered GBDT Via Back Propagation](https://mlsanity.com/paper/2109.11863/) (2021)
- [Biologically Plausible Training of Deep Neural Networks Using a Top-down Credit Assignment Network](https://mlsanity.com/paper/2208.01416/) (2022)
- [BP(λ): Online Learning via Synthetic Gradients](https://mlsanity.com/paper/2401.07044/) (2024)

## Abstract (arXiv)

When training neural networks, the use of Synthetic Gradients (SG) allows layers or modules to be trained without update locking - without waiting for a true error gradient to be backpropagated - resulting in Decoupled Neural Interfaces (DNIs). This unlocked ability of being able to update parts of a neural network asynchronously and with only local information was demonstrated to work empirically in Jaderberg et al (2016). However, there has been very little demonstration of what changes DNIs and SGs impose from a functional, representational, and learning dynamics point of view. In this paper, we study DNIs through the use of synthetic gradients on feed-forward networks to better understand their behaviour and elucidate their effect on optimisation. We show that the incorporation of SGs does not affect the representational strength of the learning system for a neural network, and prove the convergence of the learning system for linear and deep linear models. On practical problems we investigate the mechanism by which synthetic gradient estimators approximate the true loss, and, surprisingly, how that leads to drastically different layer-wise representations. Finally, we also expose the relationship of using synthetic gradients to other error approximation techniques and find a unifying language for discussion and comparison.