mlsanityData to 7 Oct 2026

Optimization without Backpropagation

arXiv 2209.06302 ↗First posted 13 Sep 2022cs.LG math.OCcited by 11

Not on the map. Why?

Abstract (arXiv)

Forward gradients have been recently introduced to bypass backpropagation in autodifferentiation, while retaining unbiased estimators of true gradients. We derive an optimality condition to obtain best approximating forward gradients, which leads us to mathematical insights that suggest optimization in high dimension is challenging with forward gradients. Our extensive experiments on test functions support this claim.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 7 of its 19 references that are in the map. First posted on arXiv 13 Sep 2022.

Builds on

7 of 19 references.

References · 7

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2015Automatic differentiation in machine learning: a surveycited by 846
  3. 2020AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradientscited by 282
  4. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  5. 2017Dissecting Adam: The Sign, Magnitude and Variance of Stochastic Gradientscited by 136
  6. 2022Gradients without Backpropagationcited by 73
  7. 2017Best practices for comparing optimization algorithmscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 5; 2025: 2; 2026: 1.

11 papers.

  1. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  2. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  3. 2023Can Forward Gradient Match Backpropagation?cited by 26
  4. 2023How to guess a gradientcited by 7
  5. 2024Beyond Backpropagation: Optimization with Multi-Tangent Forward Gradientscited by 6
  6. 2024Stepping Forward on the Last Milecited by 4
  7. 2024Accelerating Legacy Numerical Solvers by Non-intrusive Gradient-based Meta-solvingcited by 2
  8. 2024Noise-based Local Learning using Stochastic Magnetic Tunnel Junctionscited by 2
  9. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  10. 2025Warming Up for Zeroth-Order Federated Pre-Training with Low Resource Clientscited by 0
  11. 2025Forward Only Learning for Orthogonal Neural Networks of any Depthcited by 0