# Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling

arXiv: https://arxiv.org/abs/2411.17567 · HTML: https://mlsanity.com/paper/2411.17567/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 38 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 18 of 18)

- [Deep Learning in Spiking Neural Networks](https://mlsanity.com/paper/1804.08150/) (2018)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Optimal rates for zero-order convex optimization: the power of two function evaluations](https://mlsanity.com/paper/1312.2139/) (2013)
- [A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learning](https://mlsanity.com/paper/2006.06224/) (2020)
- [Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks](https://mlsanity.com/paper/1802.03796/) (2018)
- [Learning Single-Index Models with Shallow Neural Networks](https://mlsanity.com/paper/2210.15651/) (2022)
- [Scaling Forward Gradient With Local Losses](https://mlsanity.com/paper/2210.03310/) (2022)
- [SGD without Replacement: Sharper Rates for General Smooth Convex Functions](https://mlsanity.com/paper/1903.01463/) (2019)
- [Learning ReLUs via Gradient Descent](https://mlsanity.com/paper/1705.04591/) (2017)
- [Neural network learns low-dimensional polynomials with SGD near the information-theoretic limit](https://mlsanity.com/paper/2406.01581/) (2024)
- [Theory of Curriculum Learning, with Convex Loss Functions](https://mlsanity.com/paper/1812.03472/) (2018)
- [SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochs](https://mlsanity.com/paper/2107.05074/) (2021)
- [On the Statistical Benefits of Curriculum Learning](https://mlsanity.com/paper/2111.07126/) (2021)
- [Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Models](https://mlsanity.com/paper/2409.07434/) (2024)
- [Interpreting learning in biological neural networks as zero-order optimization method](https://mlsanity.com/paper/2301.11777/) (2023)
- [Convergence guarantees for forward gradient descent in the linear regression model](https://mlsanity.com/paper/2309.15001/) (2023)
- [Is Learning in Biological Neural Networks based on Stochastic Gradient Descent? An analysis using stochastic processes](https://mlsanity.com/paper/2309.05102/) (2023)
- [Hebbian learning inspired estimation of the linear regression parameters from queries](https://mlsanity.com/paper/2311.03483/) (2023)

## Built on by (most cited first, 1 of 1)

- [Adaptive directional gradients for parameterised quantum circuits](https://mlsanity.com/paper/2606.09734/) (2026)

## Abstract (arXiv)

Forward gradient descent (FGD) has been proposed as a biologically more plausible alternative of gradient descent as it can be computed without backward pass. Considering the linear model with d parameters, previous work has found that the prediction error of FGD is, however, by a factor d slower than the prediction error of stochastic gradient descent (SGD). In this paper we show that by computing ℓ FGD steps based on each training sample, this suboptimality factor becomes d/(ℓ wedge d) and thus the suboptimality of the rate disappears if ℓ gtrsim d. We also show that FGD with repeated sampling can adapt to low-dimensional structure in the input distribution. The main mathematical challenge lies in controlling the dependencies arising from the repeated sampling process.