# Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks

arXiv: https://arxiv.org/abs/2306.01870 · HTML: https://mlsanity.com/paper/2306.01870/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 44 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [signSGD: Compressed Optimisation for Non-Convex Problems](https://mlsanity.com/paper/1802.04434/) (2018)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [Directional convergence and alignment in deep learning](https://mlsanity.com/paper/2006.06657/) (2020)
- [Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced](https://mlsanity.com/paper/1806.00900/) (2018)
- [Difference Target Propagation](https://mlsanity.com/paper/1412.7525/) (2014)
- [Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures](https://mlsanity.com/paper/1807.04587/) (2018)
- [Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias](https://mlsanity.com/paper/2110.13905/) (2021)
- [How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation](https://mlsanity.com/paper/1407.7906/) (2014)
- [Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs](https://mlsanity.com/paper/2206.00939/) (2022)
- [How Important is Weight Symmetry in Backpropagation?](https://mlsanity.com/paper/1510.05067/) (2015)
- [A Theoretical Framework for Target Propagation](https://mlsanity.com/paper/2006.14331/) (2020)
- [Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architectures](https://mlsanity.com/paper/2006.12878/) (2020)
- [Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data](https://mlsanity.com/paper/2210.07082/) (2022)
- [Towards Scaling Difference Target Propagation by Learning Backprop Targets](https://mlsanity.com/paper/2201.13415/) (2022)
- [Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descent](https://mlsanity.com/paper/2106.13792/) (2021)
- [Capturing the objects of vision with neural networks](https://mlsanity.com/paper/2109.03351/) (2021)
- [Two Routes to Scalable Credit Assignment without Weight Symmetry](https://mlsanity.com/paper/2003.01513/) (2020)
- [Convergence and Alignment of Gradient Descent with Random Backpropagation Weights](https://mlsanity.com/paper/2106.06044/) (2021)
- [How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective](https://mlsanity.com/paper/2106.08453/) (2021)
- [Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithms](https://mlsanity.com/paper/2108.13446/) (2021)
- [Differentially Private Deep Learning with Direct Feedback Alignment](https://mlsanity.com/paper/2010.03701/) (2020)

## Built on by (most cited first, 1 of 1)

- [What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility](https://mlsanity.com/paper/2606.21126/) (2026)

## Abstract (arXiv)

Feedback Alignment (FA) methods are biologically inspired local learning rules for training neural networks with reduced communication between layers. While FA has potential applications in distributed and privacy-aware ML, limitations in multi-class classification and lack of theoretical understanding of the alignment mechanism have constrained its impact. This study introduces a unified framework elucidating the operational principles behind alignment in FA. Our key contributions include: (1) a novel conservation law linking changes in synaptic weights to implicit regularization that maintains alignment with the gradient, with support from experiments, (2) sufficient conditions for convergence based on the concept of alignment dominance, and (3) empirical analysis showing better alignment can enhance FA performance on complex multi-class tasks. Overall, these theoretical and practical advancements improve interpretability of bio-plausible learning rules and provide groundwork for developing enhanced FA algorithms.