mlsanityData to 14 Oct 2026

How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective

arXiv 2106.08453 ↗First posted 15 Jun 2021cs.LG stat.MLcited by 13

Not on the map. Why?

Abstract (arXiv)

Recent works have examined theoretical and empirical properties of wide neural networks trained in the Neural Tangent Kernel (NTK) regime. Given that biological neural networks are much wider than their artificial counterparts, we consider NTK regime wide neural networks as a possible model of biological neural networks. Leveraging NTK theory, we show theoretically that gradient descent drives layerwise weight updates that are aligned with their input activity correlations weighted by error, and demonstrate empirically that the result also holds in finite-width wide networks. The alignment result allows us to formulate a family of biologically-motivated, backpropagation-free learning rules that are theoretically equivalent to backpropagation in infinite-width networks. We test these learning rules on benchmark problems in feedforward and recurrent neural networks and demonstrate, in wide networks, comparable performance to backpropagation. The proposed rules are particularly effective in low data regimes, which are common in biological learning settings.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 32 of its 47 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

32 of 47 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2014How transferable are features in deep neural networks?cited by 1,225
  6. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  7. 2017Deep Neural Networks as Gaussian Processescited by 622
  8. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  9. 2018Deep Learning for Classical Japanese Literaturecited by 422
  10. 2018Gaussian Process Behaviour in Wide Deep Neural Networkscited by 317
  11. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  12. 2020Feature Learning in Infinite-Width Neural Networkscited by 190
20 more
  1. 2018A mathematical theory of semantic development in deep neural networkscited by 186
  2. 2018Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processescited by 167
  3. 2020Finite Versus Infinite Neural Networks: an Empirical Studycited by 157
  4. 2019Training Neural Networks with Local Error Signalscited by 136
  5. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 101
  6. 2020Predictive Coding Approximates Backprop along Arbitrary Computation Graphscited by 90
  7. 2019Deep Learning without Weight Transportcited by 87
  8. 2019Harnessing the Power of Infinitely Wide Deep Nets on Small-data Taskscited by 87
  9. 2015Towards Biologically Plausible Deep Learningcited by 66
  10. 2020What Do Neural Networks Learn When Trained With Random Labels?cited by 59
  11. 2015How Important is Weight Symmetry in Backpropagation?cited by 58
  12. 2020A Theoretical Framework for Target Propagationcited by 57
  13. 2020Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architecturescited by 56
  14. 2020The Neural Coding Framework for Learning Generative Modelscited by 44
  15. 2018Biologically Motivated Algorithms for Propagating Local Target Representationscited by 43
  16. 2019A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networkscited by 38
  17. 2018Biologically-plausible learning algorithms can scale to large datasetscited by 37
  18. 2018Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representationscited by 32
  19. 2021Convergence and Alignment of Gradient Descent with Random Backpropagation Weightscited by 14
  20. 2018Conducting Credit Assignment by Aligning Local Representationscited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 4; 2024: 2; 2025: 3; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 114
  2. 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
  3. 2023Beyond Geometry: Comparing the Temporal Structure of Computation in Neural Circuits with Dynamical Similarity Analysiscited by 23
  4. 2022The Influence of Learning Rule on Representation Dynamics in Wide Neural Networkscited by 19
  5. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 8
  6. 2025SineProject: Machine Unlearning for Stable Vision Language Alignmentcited by 3
  7. 2023Polarity is all you need to learn and transfer fastercited by 2
  8. 2025Curl Descent: Non-Gradient Learning Dynamics with Sign-Diverse Plasticitycited by 2
  9. 2026Overcoming Rank Collapse in Feedback Alignmentcited by 2
  10. 2023Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networkscited by 1
  11. 2026Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpoweredcited by 1
  12. 2024Training neural networks without backpropagation using particlescited by 0
1 more
  1. 2026Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Samplingcited by 0