mlsanityData to 7 Oct 2026

Can Forward Gradient Match Backpropagation?

arXiv 2306.06968 ↗First posted 12 Jun 2023cs.LG cs.CV cs.NEcited by 26

Not on the map. Why?

Abstract (arXiv)

Forward Gradients - the idea of using directional derivatives in forward differentiation mode - have recently been shown to be utilizable for neural network training while avoiding problems generally associated with backpropagation gradient computation, such as locking and memorization requirements. The cost is the requirement to guess the step direction, which is hard in high dimensions. While current solutions rely on weighted averages over isotropic guess vector distributions, we propose to strongly bias our gradient guesses in directions that are much more promising, such as feedback obtained from small, local auxiliary networks. For a standard computer vision neural network, we conduct a rigorous study systematically covering a variety of combinations of gradient targets and gradient guesses, including those previously presented in the literature. We find that using gradients obtained from a local loss as a candidate direction drastically improves on random noise in Forward Gradient methods.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 23 of its 29 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

23 of 29 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  4. 2015Automatic differentiation in machine learning: a surveycited by 846
  5. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  6. 2017A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasetscited by 374
  7. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  8. 2018i-RevNet: Deep Invertible Networkscited by 137
  9. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 135
  10. 2019Training Neural Networks with Local Error Signalscited by 135
  11. 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
  12. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 100
11 more
  1. 2019Putting An End to End-to-End: Gradient-Isolated Learning of Representationscited by 85
  2. 2022Gradients without Backpropagationcited by 73
  3. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  4. 2022Scaling Forward Gradient With Local Lossescited by 65
  5. 2015How Important is Weight Symmetry in Backpropagation?cited by 57
  6. 2020Interlocking Backpropagation: Improving depthwise model-parallelismcited by 17
  7. 2017Building a Regular Decision Boundary with Deep Networkscited by 11
  8. 2022Optimization without Backpropagationcited by 11
  9. 2021Decoupled Greedy Learning of CNNs for Synchronous and Asynchronous Distributed Learningcited by 6
  10. 2022Local Learning on Transformers via Feature Reconstructioncited by 4
  11. 2023Local Learning with Neuron Groupscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 12; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024Scaling Supervised Local Learning with Augmented Auxiliary Networkscited by 8
  2. 2023How to guess a gradientcited by 7
  3. 2024PETRA: Parallel End-to-end Training with Reversible Architecturescited by 6
  4. 2024Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradientscited by 6
  5. 2025Learning long range dependencies through time reversal symmetry breakingcited by 6
  6. 2024Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reductioncited by 5
  7. 2024Second-Order Forward-Mode Automatic Differentiation for Optimizationcited by 5
  8. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  9. 2024Stepping Forward on the Last Milecited by 4
  10. 2025TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Modelscited by 4
  11. 2024Towards training digitally-tied analog blocks via hybrid gradient computationcited by 3
  12. 2024Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solvingcited by 2
14 more
  1. 2025FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivativecited by 2
  2. 2025An Affective-Taxis Hypothesis for Alignment and Interpretabilitycited by 1
  3. 2025Memory Savings at What Cost? A Study of Alternatives to Backpropagationcited by 1
  4. 2024Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networkscited by 0
  5. 2025Noise-based reward-modulated learningcited by 0
  6. 2025Backpropagation-Free Metropolis-Adjusted Langevin Algorithmcited by 0
  7. 2025Low-rank surrogate modeling and stochastic zero-order optimization for training of neural networks with black-box layerscited by 0
  8. 2025Local Reinforcement Learning with Action-Conditioned Root Mean Squared Q-Functionscited by 0
  9. 2025A Unified Perspective on Optimization in Machine Learning and Neuroscience: From Gradient Descent to Neural Adaptationcited by 0
  10. 2025Towards Scalable Backpropagation-Free Gradient Estimationcited by 0
  11. 2025Forward Only Learning for Orthogonal Neural Networks of any Depthcited by 0
  12. 2026Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagationcited by 0
  13. 2026Scalar Federated Learning for Linear Quadratic Regulatorcited by 0
  14. 2026Adaptive directional gradients for parameterised quantum circuitscited by 0