# Principled Training of Neural Networks with Direct Feedback Alignment

arXiv: https://arxiv.org/abs/1906.04554 · HTML: https://mlsanity.com/paper/1906.04554/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 23 of 23)

- [Credit Assignment in Neural Networks through Deep Feedback Control](https://mlsanity.com/paper/2106.07887/) (2021)
- [Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence](https://mlsanity.com/paper/2106.01288/) (2021)
- [Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment](https://mlsanity.com/paper/2312.09257/) (2023)
- [Applications of the Free Energy Principle to Machine Learning and Neuroscience](https://mlsanity.com/paper/2107.00140/) (2021)
- [Adversarial Robustness by Design through Analog Computing and Synthetic Gradients](https://mlsanity.com/paper/2101.02115/) (2021)
- [Low-Variance Forward Gradients using Direct Feedback Alignment and Momentum](https://mlsanity.com/paper/2212.07282/) (2022)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Scaling Laws Beyond Backpropagation](https://mlsanity.com/paper/2210.14593/) (2022)
- [Training Spiking Neural Networks via Augmented Direct Feedback Alignment](https://mlsanity.com/paper/2409.07776/) (2024)
- [Training Large Neural Networks With Low-Dimensional Error Feedback](https://mlsanity.com/paper/2502.20580/) (2025)
- [HCL-FF: Hierarchical and Contrastive Learning for Forward-Forward Algorithm](https://mlsanity.com/paper/2605.24797/) (2026)
- [Overcoming Rank Collapse in Feedback Alignment](https://mlsanity.com/paper/2606.11123/) (2026)
- [Convergence Analysis and Implicit Regularization of Feedback Alignment for Deep Linear Networks](https://mlsanity.com/paper/2110.10815/) (2021)
- [Forward Learning of Graph Neural Networks](https://mlsanity.com/paper/2403.11004/) (2024)
- [Asymptotically Stable Quaternion-valued Hopfield-structured Neural Network with Periodic Projection-based Supervised Learning Rules](https://mlsanity.com/paper/2510.16607/) (2025)
- [Diffusing Blame: Task-Dependent Credit Assignment in Biologically Plausible Dual-Stream Networks](https://mlsanity.com/paper/2606.31700/) (2026)
- [Learning with augmented target information: An alternative theory of Feedback Alignment](https://mlsanity.com/paper/2304.01406/) (2023)
- [Random Feedback Alignment Algorithms to train Neural Networks: Why do they Align?](https://mlsanity.com/paper/2306.02325/) (2023)
- [Can Local Representation Alignment RNNs Solve Temporal Tasks?](https://mlsanity.com/paper/2504.13531/) (2025)
- [Forward Only Learning for Orthogonal Neural Networks of any Depth](https://mlsanity.com/paper/2512.20668/) (2025)
- [Score Broadcast and Decorrelation: A General Framework for Broadcast-Based Credit Assignment](https://mlsanity.com/paper/2605.30638/) (2026)
- [What Accuracy and Gradient Cosine Miss: Evaluating Feedback Alignment via Scale Stability, Reference Validity, and Depth Utility](https://mlsanity.com/paper/2606.21126/) (2026)
- [Common-Mode Collapse and Recovery in Direct Feedback Alignment](https://mlsanity.com/paper/2609.31589/) (2026)

## Abstract (arXiv)

The backpropagation algorithm has long been the canonical training method for neural networks. Modern paradigms are implicitly optimized for it, and numerous guidelines exist to ensure its proper use. Recently, synthetic gradients methods -where the error gradient is only roughly approximated - have garnered interest. These methods not only better portray how biological brains are learning, but also open new computational possibilities, such as updating layers asynchronously. Even so, they have failed to scale past simple tasks like MNIST or CIFAR-10. This is in part due to a lack of standards, leading to ill-suited models and practices forbidding such methods from performing to the best of their abilities. In this work, we focus on direct feedback alignment and present a set of best practices justified by observations of the alignment angles. We characterize a bottleneck effect that prevents alignment in narrow layers, and hypothesize it may explain why feedback alignment methods have yet to scale to large convolutional networks.