mlsanityData to 7 Oct 2026

Convergence and Alignment of Gradient Descent with Random Backpropagation Weights

arXiv 2106.06044 ↗First posted 10 Jun 2021stat.ML cs.LGcited by 14

Not on the map. Why?

Abstract (arXiv)

Stochastic gradient descent with backpropagation is the workhorse of artificial neural networks. It has long been recognized that backpropagation fails to be a biologically plausible algorithm. Fundamentally, it is a non-local procedure -- updating one neuron's synaptic weights requires knowledge of synaptic weights or receptive fields of downstream neurons. This limits the use of artificial neural networks as a tool for understanding the biological principles of information processing in the brain. Lillicrap et al. (2016) propose a more biologically plausible "feedback alignment" algorithm that uses random and fixed backpropagation weights, and show promising simulations. In this paper we study the mathematical properties of the feedback alignment procedure by analyzing convergence and alignment for two-layer networks under squared error loss. In the overparameterized setting, we prove that the error converges to zero exponentially fast, and also that regularization is necessary in order for the parameters to become aligned with the random backpropagation weights. Simulations are given that are consistent with this analysis and suggest further generalizations. These results contribute to our understanding of how biologically plausible algorithms might carry out weight learning in a manner different from Hebbian learning, with performance that is comparable with the full non-local backpropagation algorithm.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 16 of its 37 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

16 of 37 references.

References · 16

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  4. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  5. 2013Risk-sensitive Reinforcement Learningcited by 129
  6. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 100
  7. 2013Dropout Training as Adaptive Regularizationcited by 89
  8. 2019Deep Learning without Weight Transportcited by 87
  9. 2020Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHScited by 74
  10. 2020Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regretcited by 57
  11. 2020Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architecturescited by 56
  12. 2020Meta-Learning through Hebbian Plasticity in Random Networkscited by 47
4 more
  1. 2017Global Guarantees for Enforcing Deep Generative Priors by Empirical Riskcited by 37
  2. 2019Biologically inspired alternatives to backpropagation through time for learning in recurrent neural netscited by 20
  3. 2016Towards deep learning with spiking neurons in energy based models with contrastive Hebbian plasticitycited by 4
  4. 2020Model Repair: Robust Recovery of Over-Parameterized Statistical Modelscited by 2

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 4; 2023: 5; 2024: 2.

14 papers.

Sort:

  1. 2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
  2. 2022The Influence of Learning Rule on Representation Dynamics in Wide Neural Networkscited by 19
  3. 2021How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspectivecited by 13
  4. 2021How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorizationcited by 5
  5. 2024LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronizationcited by 5
  6. 2023Finding Regularized Competitive Equilibria of Heterogeneous Agent Macroeconomic Models with Reinforcement Learningcited by 4
  7. 2022Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rulescited by 3
  8. 2022Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learningcited by 2
  9. 2023Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networkscited by 1
  10. 2022Emergent organization of receptive fields in networks of excitatory and inhibitory neuronscited by 0
  11. 2023A method for the ethical analysis of brain-inspired AIcited by 0
  12. 2023Random Feedback Alignment Algorithms to train Neural Networks: Why do they Align?cited by 0
2 more
  1. 2023Biologically-Motivated Learning Model for Instructed Visual Processingcited by 0
  2. 2024Taming Equilibrium Bias in Risk-Sensitive Multi-Agent Reinforcement Learningcited by 0