# Convergence and Alignment of Gradient Descent with Random Backpropagation Weights

arXiv: https://arxiv.org/abs/2106.06044 · HTML: https://mlsanity.com/paper/2106.06044/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 37 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 16 of 16)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Risk-sensitive Reinforcement Learning](https://mlsanity.com/paper/1311.2097/) (2013)
- [Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures](https://mlsanity.com/paper/1807.04587/) (2018)
- [Dropout Training as Adaptive Regularization](https://mlsanity.com/paper/1307.1493/) (2013)
- [Deep Learning without Weight Transport](https://mlsanity.com/paper/1904.05391/) (2019)
- [Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS](https://mlsanity.com/paper/2009.10683/) (2020)
- [Risk-Sensitive Reinforcement Learning: Near-Optimal Risk-Sample Tradeoff in Regret](https://mlsanity.com/paper/2006.13827/) (2020)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [Meta-Learning through Hebbian Plasticity in Random Networks](https://mlsanity.com/paper/2007.02686/) (2020)
- [Global Guarantees for Enforcing Deep Generative Priors by Empirical Risk](https://mlsanity.com/paper/1705.07576/) (2017)
- [Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets](https://mlsanity.com/paper/1901.09049/) (2019)
- [Towards deep learning with spiking neurons in energy based models with contrastive Hebbian plasticity](https://mlsanity.com/paper/1612.03214/) (2016)
- [Model Repair: Robust Recovery of Over-Parameterized Statistical Models](https://mlsanity.com/paper/2005.09912/) (2020)

## Built on by (most cited first, 14 of 14)

- [Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2111.03947/) (2021)
- [The Influence of Learning Rule on Representation Dynamics in Wide Neural Networks](https://mlsanity.com/paper/2210.02157/) (2022)
- [How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective](https://mlsanity.com/paper/2106.08453/) (2021)
- [How and When Random Feedback Works: A Case Study of Low-Rank Matrix Factorization](https://mlsanity.com/paper/2111.08706/) (2021)
- [LLS: Local Learning Rule for Deep Neural Networks Inspired by Neural Activity Synchronization](https://mlsanity.com/paper/2405.15868/) (2024)
- [Finding Regularized Competitive Equilibria of Heterogeneous Agent Macroeconomic Models with Reinforcement Learning](https://mlsanity.com/paper/2303.04833/) (2023)
- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)
- [Cascaded Gaps: Towards Gap-Dependent Regret for Risk-Sensitive Reinforcement Learning](https://mlsanity.com/paper/2203.03110/) (2022)
- [Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks](https://mlsanity.com/paper/2306.01870/) (2023)
- [Emergent organization of receptive fields in networks of excitatory and inhibitory neurons](https://mlsanity.com/paper/2205.13614/) (2022)
- [A method for the ethical analysis of brain-inspired AI](https://mlsanity.com/paper/2305.10938/) (2023)
- [Random Feedback Alignment Algorithms to train Neural Networks: Why do they Align?](https://mlsanity.com/paper/2306.02325/) (2023)
- [Biologically-Motivated Learning Model for Instructed Visual Processing](https://mlsanity.com/paper/2306.02415/) (2023)
- [Taming Equilibrium Bias in Risk-Sensitive Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.02724/) (2024)

## Abstract (arXiv)

Stochastic gradient descent with backpropagation is the workhorse of artificial neural networks. It has long been recognized that backpropagation fails to be a biologically plausible algorithm. Fundamentally, it is a non-local procedure -- updating one neuron's synaptic weights requires knowledge of synaptic weights or receptive fields of downstream neurons. This limits the use of artificial neural networks as a tool for understanding the biological principles of information processing in the brain. Lillicrap et al. (2016) propose a more biologically plausible "feedback alignment" algorithm that uses random and fixed backpropagation weights, and show promising simulations. In this paper we study the mathematical properties of the feedback alignment procedure by analyzing convergence and alignment for two-layer networks under squared error loss. In the overparameterized setting, we prove that the error converges to zero exponentially fast, and also that regularization is necessary in order for the parameters to become aligned with the random backpropagation weights. Simulations are given that are consistent with this analysis and suggest further generalizations. These results contribute to our understanding of how biologically plausible algorithms might carry out weight learning in a manner different from Hebbian learning, with performance that is comparable with the full non-local backpropagation algorithm.