# Convergence guarantees for forward gradient descent in the linear regression model

arXiv: https://arxiv.org/abs/2309.15001 · HTML: https://mlsanity.com/paper/2309.15001/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2023.

## Built on by (most cited first, 2 of 2)

- [Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling](https://mlsanity.com/paper/2411.17567/) (2024)
- [Adaptive directional gradients for parameterised quantum circuits](https://mlsanity.com/paper/2606.09734/) (2026)

## Abstract (arXiv)

Renewed interest in the relationship between artificial and biological neural networks motivates the study of gradient-free methods. Considering the linear regression model with random design, we theoretically analyze in this work the biologically motivated (weight-perturbed) forward gradient scheme that is based on random linear combination of the gradient. If d denotes the number of parameters and k the number of samples, we prove that the mean squared error of this method converges for kgtrsim d2log(d) with rate d2log(d)/k. Compared to the dimension dependence d for stochastic gradient descent, an additional factor dlog(d) occurs.