mlsanityData to 7 Oct 2026

Gradients without Backpropagation

arXiv 2202.08587 ↗First posted 17 Feb 2022cs.LG stat.MLcited by 73

Not on the map. Why?

Abstract (arXiv)

Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation, is a special case within the general family of automatic differentiation algorithms that also includes the forward mode. We present a method to compute gradients based solely on the directional derivative that one can compute exactly and efficiently via the forward mode. We call this formulation the forward gradient, an unbiased estimate of the gradient that can be evaluated in a single forward run of the function, entirely eliminating the need for backpropagation in gradient descent. We demonstrate forward gradient descent in a range of problems, showing substantial savings in computation and enabling training up to twice as fast in some cases.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 17 of its 45 references that are in the map. First posted on arXiv 17 Feb 2022.

Builds on

17 of 45 references.

References · 17

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  4. 2015Automatic differentiation in machine learning: a surveycited by 846
  5. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  6. 2015Coordinate Descent Algorithmscited by 218
  7. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 135
  8. 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 77
  9. 2015Towards Biologically Plausible Deep Learningcited by 66
  10. 2020A Theoretical Framework for Target Propagationcited by 56
  11. 2018Stochastic Quasi-Gradient Methods: Variance Reduction via Jacobian Sketchingcited by 34
  12. 2018SEGA: Variance Reduction via Gradient Sketchingcited by 29
5 more
  1. 2020Deriving Differential Target Propagation from Iterating Approximate Inversescited by 15
  2. 2018Accelerated Stochastic Matrix Inversion: General Theory and Speeding up BFGS Rules for Faster Second-Order Optimizationcited by 11
  3. 2020Langevin Monte Carlo: random coordinate descent and variance reductioncited by 7
  4. 2017Divide-and-Conquer Checkpointing for Arbitrary Programs with No User Annotationcited by 6
  5. 2018An Accelerated Directional Derivative Method for Smooth Stochastic Convex Optimizationcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 16; 2024: 23; 2025: 19; 2026: 11.

73 papers.

Sort:

By area:

  1. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  2. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  3. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  4. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  5. 2022Scaling Forward Gradient With Local Lossescited by 65
  6. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  7. 2023Can Forward Gradient Match Backpropagation?cited by 26
  8. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  9. 2023Online learning of long-range dependenciescited by 25
  10. 2023FwdLLM: Efficient FedLLM using Forward Gradientcited by 25
  11. 2023Training Convolutional Neural Networks with the Forward-Forward algorithmcited by 13
  12. 2023Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networkscited by 12
38 more
  1. 2022Optimization without Backpropagationcited by 11
  2. 2024Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimizationcited by 11
  3. 2024Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Modelcited by 11
  4. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  5. 2024End-to-End Training Induces Information Bottleneck through Layer-Role Differentiation: A Comparative Analysis with Layer-wise Trainingcited by 8
  6. 2024Scaling Supervised Local Learning with Augmented Auxiliary Networkscited by 8
  7. 2025NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagationcited by 8
  8. 2022Low-Variance Forward Gradients using Direct Feedback Alignment and Momentumcited by 7
  9. 2023How to guess a gradientcited by 7
  10. 2024Thinking Forward: Memory-Efficient Federated Finetuning of Language Modelscited by 7
  11. 2023Smoothing Methods for Automatic Differentiation Across Conditional Branchescited by 6
  12. 2024Gradient-Free Training of Recurrent Neural Networks using Random Perturbationscited by 6
  13. 2024FFCL: Forward-Forward Net with Cortical Loops, Training and Inference on Edge Without Backpropagationcited by 6
  14. 2024Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradientscited by 6
  15. 2023Extending the Forward Forward Algorithmcited by 5
  16. 2024Projected Forward Gradient-Guided Frank-Wolfe Algorithm via Variance Reductioncited by 5
  17. 2024Second-Order Forward-Mode Automatic Differentiation for Optimizationcited by 5
  18. 2024Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approachcited by 5
  19. 2023The Integrated Forward-Forward Algorithm: Integrating Forward-Forward and Shallow Backpropagation With Local Lossescited by 4
  20. 2024Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversioncited by 4
  21. 2024Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learningcited by 4
  22. 2024Towards Interpretable Deep Local Learning with Successive Gradient Reconciliationcited by 4
  23. 2024Stepping Forward on the Last Milecited by 4
  24. 2025TITAN-Guide: Taming Inference-Time AligNment for Guided Text-to-Video Diffusion Modelscited by 4
  25. 2023Randomized Forward Mode of Automatic Differentiation For Optimization Algorithmscited by 3
  26. 2024FLOPS: Forward Learning with OPtimal Samplingcited by 3
  27. 2023Node Perturbation Can Effectively Train Multi-Layer Neural Networkscited by 2
  28. 2024Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solvingcited by 2
  29. 2024Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networkscited by 2
  30. 2024Noise-based Local Learning using Stochastic Magnetic Tunnel Junctionscited by 2
  31. 2025Gradient Prediction with Control Variates in the Cheap-Forward Regimecited by 2
  32. 2025FedMPDD: Communication-Efficient Federated Learning with Privacy Preservation Attributes via Projected Directional Derivativecited by 2
  33. 2023Leader-Follower Neural Networks with Local Error Signals Inspired by Complex Collectivescited by 1
  34. 2023Hebbian learning inspired estimation of the linear regression parameters from queriescited by 1
  35. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  36. 2025Memory-Efficient LLM Training by Various-Grained Low-Rank Projection of Gradientscited by 1
  37. 2025Memory Savings at What Cost? A Study of Alternatives to Backpropagationcited by 1
  38. 2025More Than Memory Savings: Zeroth-Order Optimization Mitigates Forgetting in Continual Learningcited by 1

50 most cited of 73 All 73 (JSON)