# Can Forward Gradient Match Backpropagation?

arXiv: https://arxiv.org/abs/2306.06968 · HTML: https://mlsanity.com/paper/2306.06968/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 12 Jun 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets](https://mlsanity.com/paper/1707.08819/) (2017)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Decoupled Neural Interfaces using Synthetic Gradients](https://mlsanity.com/paper/1608.05343/) (2016)
- [i-RevNet: Deep Invertible Networks](https://mlsanity.com/paper/1802.07088/) (2018)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Greedy Layerwise Learning Can Scale to ImageNet](https://mlsanity.com/paper/1812.11446/) (2018)
- [Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures](https://mlsanity.com/paper/1807.04587/) (2018)
- [Putting An End to End-to-End: Gradient-Isolated Learning of Representations](https://mlsanity.com/paper/1905.11786/) (2019)
- [Gradients without Backpropagation](https://mlsanity.com/paper/2202.08587/) (2022)
- [Revisiting Locally Supervised Learning: an Alternative to End-to-end Training](https://mlsanity.com/paper/2101.10832/) (2021)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [How Important is Weight Symmetry in Backpropagation?](https://mlsanity.com/paper/1510.05067/) (2015)
- [Interlocking Backpropagation: Improving depthwise model-parallelism](https://mlsanity.com/paper/2010.04116/) (2020)
- [Building a Regular Decision Boundary with Deep Networks](https://mlsanity.com/paper/1703.01775/) (2017)
- [Optimization without Backpropagation](https://mlsanity.com/paper/2209.06302/) (2022)
- [Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learning](https://mlsanity.com/paper/2106.06401/) (2021)
- [Local Learning on Transformers via Feature Reconstruction](https://mlsanity.com/paper/2212.14215/) (2022)
- [Local Learning with Neuron Groups](https://mlsanity.com/paper/2301.07635/) (2023)

## Built on by (most cited first, 26 of 26)

- [Scaling Supervised Local Learning with Augmented Auxiliary Networks](https://mlsanity.com/paper/2402.17318/) (2024)
- [How to guess a gradient](https://mlsanity.com/paper/2312.04709/) (2023)
- [Learning long range dependencies through time reversal symmetry breaking](https://mlsanity.com/paper/2506.05259/) (2025)
- [PETRA: Parallel End-to-end Training with Reversible Architectures](https://mlsanity.com/paper/2406.02052/) (2024)
- [Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients](https://mlsanity.com/paper/2410.17764/) (2024)
- [Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reduction](https://mlsanity.com/paper/2403.12511/) (2024)
- [Second-Order Forward-Mode Automatic Differentiation for Optimization](https://mlsanity.com/paper/2408.10419/) (2024)
- [Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach](https://mlsanity.com/paper/2411.05873/) (2024)
- [Towards training digitally-tied analog blocks via hybrid gradient computation](https://mlsanity.com/paper/2409.03306/) (2024)
- [Stepping Forward on the Last Mile](https://mlsanity.com/paper/2411.04036/) (2024)
- [TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Models](https://mlsanity.com/paper/2508.00289/) (2025)
- [Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solving](https://mlsanity.com/paper/2405.02952/) (2024)
- [FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivative](https://mlsanity.com/paper/2512.20814/) (2025)
- [An Affective-Taxis Hypothesis for Alignment and Interpretability](https://mlsanity.com/paper/2505.17024/) (2025)
- [Memory Savings at What Cost? A Study of Alternatives to Backpropagation](https://mlsanity.com/paper/2506.21833/) (2025)
- [Low-rank surrogate modeling and stochastic zero-order optimization for training of neural networks with black-box layers](https://mlsanity.com/paper/2509.15113/) (2025)
- [Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks](https://mlsanity.com/paper/2403.08837/) (2024)
- [Noise-based reward-modulated learning](https://mlsanity.com/paper/2503.23972/) (2025)
- [Backpropagation-Free Metropolis-Adjusted Langevin Algorithm](https://mlsanity.com/paper/2505.18081/) (2025)
- [Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functions](https://mlsanity.com/paper/2510.06649/) (2025)
- [A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptation](https://mlsanity.com/paper/2510.18812/) (2025)
- [Towards Scalable Backpropagation-Free Gradient Estimation](https://mlsanity.com/paper/2511.03110/) (2025)
- [Forward Only Learning for Orthogonal Neural Networks of any Depth](https://mlsanity.com/paper/2512.20668/) (2025)
- [Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation](https://mlsanity.com/paper/2602.14701/) (2026)
- [Scalar Federated Learning for Linear Quadratic Regulator](https://mlsanity.com/paper/2604.05088/) (2026)
- [Adaptive directional gradients for parameterised quantum circuits](https://mlsanity.com/paper/2606.09734/) (2026)

## Abstract (arXiv)

Forward Gradients - the idea of using directional derivatives in forward differentiation mode - have recently been shown to be utilizable for neural network training while avoiding problems generally associated with backpropagation gradient computation, such as locking and memorization requirements. The cost is the requirement to guess the step direction, which is hard in high dimensions. While current solutions rely on weighted averages over isotropic guess vector distributions, we propose to strongly bias our gradient guesses in directions that are much more promising, such as feedback obtained from small, local auxiliary networks. For a standard computer vision neural network, we conduct a rigorous study systematically covering a variety of combinations of gradient targets and gradient guesses, including those previously presented in the literature. We find that using gradients obtained from a local loss as a candidate direction drastically improves on random noise in Forward Gradient methods.