# Optimization without Backpropagation

arXiv: https://arxiv.org/abs/2209.06302 · HTML: https://mlsanity.com/paper/2209.06302/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 13 Sep 2022.

## Builds on (most cited first, 7 of 7)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Automatic differentiation in machine learning: a survey](https://mlsanity.com/paper/1502.05767/) (2015)
- [AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients](https://mlsanity.com/paper/2010.07468/) (2020)
- [A Simple Convergence Proof of Adam and Adagrad](https://mlsanity.com/paper/2003.02395/) (2020)
- [Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradients](https://mlsanity.com/paper/1705.07774/) (2017)
- [Gradients without Backpropagation](https://mlsanity.com/paper/2202.08587/) (2022)
- [Best practices for comparing optimization algorithms](https://mlsanity.com/paper/1709.08242/) (2017)

## Built on by (most cited first, 11 of 11)

- [Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark](https://mlsanity.com/paper/2402.11592/) (2024)
- [DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Training](https://mlsanity.com/paper/2310.02025/) (2023)
- [Can Forward Gradient Match Backpropagation?](https://mlsanity.com/paper/2306.06968/) (2023)
- [How to guess a gradient](https://mlsanity.com/paper/2312.04709/) (2023)
- [Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradients](https://mlsanity.com/paper/2410.17764/) (2024)
- [Stepping Forward on the Last Mile](https://mlsanity.com/paper/2411.04036/) (2024)
- [Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solving](https://mlsanity.com/paper/2405.02952/) (2024)
- [Noise-based Local Learning using Stochastic Magnetic Tunnel Junctions](https://mlsanity.com/paper/2412.12783/) (2024)
- [Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered](https://mlsanity.com/paper/2605.15622/) (2026)
- [Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clients](https://mlsanity.com/paper/2509.03503/) (2025)
- [Forward Only Learning for Orthogonal Neural Networks of any Depth](https://mlsanity.com/paper/2512.20668/) (2025)

## Abstract (arXiv)

Forward gradients have been recently introduced to bypass backpropagation in autodifferentiation, while retaining unbiased estimators of true gradients. We derive an optimality condition to obtain best approximating forward gradients, which leads us to mathematical insights that suggest optimization in high dimension is challenging with forward gradients. Our extensive experiments on test functions support this claim.