Implicit Regularization in Feedback Alignment Learning Mechanisms for Neural Networks
Not on the map. Why?
Abstract (arXiv)
Feedback Alignment (FA) methods are biologically inspired local learning rules for training neural networks with reduced communication between layers. While FA has potential applications in distributed and privacy-aware ML, limitations in multi-class classification and lack of theoretical understanding of the alignment mechanism have constrained its impact. This study introduces a unified framework elucidating the operational principles behind alignment in FA. Our key contributions include: (1) a novel conservation law linking changes in synaptic weights to implicit regularization that maintains alignment with the gradient, with support from experiments, (2) sufficient conditions for convergence based on the concept of alignment dominance, and (3) empirical analysis showing better alignment can enhance FA performance on complex multi-class tasks. Overall, these theoretical and practical advancements improve interpretability of bio-plausible learning rules and provide groundwork for developing enhanced FA algorithms.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 44 references that are in the map. First posted on arXiv 2 Jun 2023.
Builds on
23 of 44 references.
References · 23
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 561
- 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
- 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
- 2020Directional convergence and alignment in deep learningcited by 172
- 2014Difference Target Propagationcited by 144
- 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 100
- 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
- 2014How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagationcited by 77
- 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
- 2015How Important is Weight Symmetry in Backpropagation?cited by 57
11 more
- 2020Direct Feedback Alignment Scales to Modern Deep Learning Tasks and Architecturescited by 56
- 2020A Theoretical Framework for Target Propagationcited by 56
- 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
- 2022Towards Scaling Difference Target Propagation by Learning Backprop Targetscited by 34
- 2021Proxy Convexity: A Unified Framework for the Analysis of Neural Networks Trained by Gradient Descentcited by 21
- 2021Capturing the objects of vision with neural networkscited by 21
- 2020Two Routes to Scalable Credit Assignment without Weight Symmetrycited by 19
- 2021Convergence and Alignment of Gradient Descent with Random Backpropagation Weightscited by 14
- 2021How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspectivecited by 13
- 2021Benchmarking the Accuracy and Robustness of Feedback Alignment Algorithmscited by 9
- 2020Differentially Private Deep Learning with Direct Feedback Alignmentcited by 7
Built on by
Citing papers per year: 2026: 1.
1 paper.