mlsanityData to 14 Oct 2026

Improving the Convergence Rates of Forward Gradient Descent with Repeated Sampling

arXiv 2411.17567 ↗First posted 26 Nov 2024math.ST cs.LG cs.NEcited by 1

Not on the map. Why?

Abstract (arXiv)

Forward gradient descent (FGD) has been proposed as a biologically more plausible alternative of gradient descent as it can be computed without backward pass. Considering the linear model with d parameters, previous work has found that the prediction error of FGD is, however, by a factor d slower than the prediction error of stochastic gradient descent (SGD). In this paper we show that by computing ℓ FGD steps based on each training sample, this suboptimality factor becomes d/(ℓ wedge d) and thus the suboptimality of the rate disappears if ℓ gtrsim d. We also show that FGD with repeated sampling can adapt to low-dimensional structure in the input distribution. The main mathematical challenge lies in controlling the dependencies arising from the repeated sampling process.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 18 of its 38 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

18 of 38 references.

References · 18

  1. 2018Deep Learning in Spiking Neural Networkscited by 263
  2. 2021Curriculum Learning: A Surveycited by 259
  3. 2013Optimal rates for zero-order convex optimization: the power of two function evaluationscited by 198
  4. 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 162
  5. 2018Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networkscited by 114
  6. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  7. 2022Scaling Forward Gradient With Local Lossescited by 65
  8. 2019SGD without Replacement: Sharper Rates for General Smooth Convex Functionscited by 61
  9. 2017Learning ReLUs via Gradient Descentcited by 54
  10. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  11. 2018Theory of Curriculum Learning, with Convex Loss Functionscited by 33
  12. 2021SGD: The Role of Implicit Regularization, Batch-size and Multiple-epochscited by 22
6 more
  1. 2021On the Statistical Benefits of Curriculum Learningcited by 10
  2. 2024Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Modelscited by 7
  3. 2023Interpreting learning in biological neural networks as zero-order optimization methodcited by 4
  4. 2023Convergence guarantees for forward gradient descent in the linear regression modelcited by 2
  5. 2023Is Learning in Biological Neural Networks based on Stochastic Gradient Descent? An analysis using stochastic processescited by 1
  6. 2023Hebbian learning inspired estimation of the linear regression parameters from queriescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Adaptive directional gradients for parameterised quantum circuitscited by 0