# Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks

arXiv: https://arxiv.org/abs/2110.10815 · HTML: https://mlsanity.com/paper/2110.10815/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 20 Oct 2021.

## Builds on (most cited first, 10 of 10)

- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [A mathematical theory of semantic development in deep neural networks](https://mlsanity.com/paper/1810.10531/) (2018)
- [A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks](https://mlsanity.com/paper/1810.02281/) (2018)
- [Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks](https://mlsanity.com/paper/1904.13262/) (2019)
- [Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures](https://mlsanity.com/paper/1807.04587/) (2018)
- [The Implicit Bias of Depth: How Incremental Learning Drives Generalization](https://mlsanity.com/paper/1909.12051/) (2019)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [Feedback alignment in deep convolutional networks](https://mlsanity.com/paper/1812.06488/) (2018)
- [Principled Training of Neural Networks with Direct Feedback Alignment](https://mlsanity.com/paper/1906.04554/) (2019)
- [Training DNNs in O(1) memory with MEM-DFA using Random Matrices](https://mlsanity.com/paper/2012.11745/) (2020)

## Built on by (most cited first, 1 of 1)

- [Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rules](https://mlsanity.com/paper/2206.00823/) (2022)

## Abstract (arXiv)

We theoretically analyze the Feedback Alignment (FA) algorithm, an efficient alternative to backpropagation for training neural networks. We provide convergence guarantees with rates for deep linear networks for both continuous and discrete dynamics. Additionally, we study incremental learning phenomena for shallow linear networks. Interestingly, certain specific initializations imply that negligible components are learned before the principal ones, thus potentially negatively affecting the effectiveness of such a learning algorithm; a phenomenon we classify as implicit anti-regularization. We also provide initialization schemes where the components of the problem are approximately learned by decreasing order of importance, thus providing a form of implicit regularization.