# Random Feedback Alignment Algorithms to train Neural Networks: Why do they Align?

arXiv: https://arxiv.org/abs/2306.02325 · HTML: https://mlsanity.com/paper/2306.02325/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 22 references that are in the map.
- First posted on arXiv 4 Jun 2023.

## Builds on (most cited first, 12 of 12)

- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [The Forward-Forward Algorithm: Some Preliminary Investigations](https://mlsanity.com/paper/2212.13345/) (2022)
- [Training Neural Networks with Local Error Signals](https://mlsanity.com/paper/1901.06656/) (2019)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [Blessing of dimensionality: mathematical foundations of the statistical physics of data](https://mlsanity.com/paper/1801.03421/) (2018)
- [Decoupled Parallel Backpropagation with Convergence Guarantee](https://mlsanity.com/paper/1804.10574/) (2018)
- [Neuromorphic Deep Learning Machines](https://mlsanity.com/paper/1612.05596/) (2016)
- [Random Walks: A Review of Algorithms and Applications](https://mlsanity.com/paper/2008.03639/) (2020)
- [Direct Feedback Alignment with Sparse Connections for Local Learning](https://mlsanity.com/paper/1903.02083/) (2019)
- [Principled Training of Neural Networks with Direct Feedback Alignment](https://mlsanity.com/paper/1906.04554/) (2019)
- [Convergence and Alignment of Gradient Descent with Random Backpropagation Weights](https://mlsanity.com/paper/2106.06044/) (2021)
- [Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms](https://mlsanity.com/paper/2108.13446/) (2021)

## Abstract (arXiv)

Feedback alignment algorithms are an alternative to backpropagation to train neural networks, whereby some of the partial derivatives that are required to compute the gradient are replaced by random terms. This essentially transforms the update rule into a random walk in weight space. Surprisingly, learning still works with those algorithms, including training of deep neural networks. This is generally attributed to an alignment of the update of the random walker with the true gradient - the eponymous gradient alignment -- which drives an approximate gradient descend. The mechanism that leads to this alignment remains unclear, however. In this paper, we use mathematical reasoning and simulations to investigate gradient alignment. We observe that the feedback alignment update rule has fixed points, which correspond to extrema of the loss function. We show that gradient alignment is a stability criterion for those fixed points. It is only a necessary criterion for algorithm performance. Experimentally, we demonstrate that high levels of gradient alignment can lead to poor algorithm performance and that the alignment is not always driving the gradient descend.