# Pretraining with Random Noise for Fast and Robust Learning without Weight Transport

arXiv: https://arxiv.org/abs/2405.16731 · HTML: https://mlsanity.com/paper/2405.16731/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 103 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 21 of 21)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Implicit Regularization in Deep Matrix Factorization](https://mlsanity.com/paper/1905.13655/) (2019)
- [Direct Feedback Alignment Provides Learning in Deep Neural Networks](https://mlsanity.com/paper/1609.01596/) (2016)
- [The Low-Rank Simplicity Bias in Deep Networks](https://mlsanity.com/paper/2103.10427/) (2021)
- [Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures](https://mlsanity.com/paper/1807.04587/) (2018)
- [How Does Information Bottleneck Help Deep Learning?](https://mlsanity.com/paper/2305.18887/) (2023)
- [Deep Learning without Weight Transport](https://mlsanity.com/paper/1904.05391/) (2019)
- [How Important is Weight Symmetry in Backpropagation?](https://mlsanity.com/paper/1510.05067/) (2015)
- [Representation Based Complexity Measures for Predicting Generalization in Deep Learning](https://mlsanity.com/paper/2012.02775/) (2020)
- [Deep learning with asymmetric connections and Hebbian updates](https://mlsanity.com/paper/1812.07965/) (2018)
- [Meta-Learning Biologically Plausible Plasticity Rules with Random Feedback Pathways](https://mlsanity.com/paper/2210.16414/) (2022)
- [An Unsupervised STDP-based Spiking Neural Network Inspired By Biologically Plausible Learning Rules and Connections](https://mlsanity.com/paper/2207.02727/) (2022)
- [Learning efficient backprojections across cortical hierarchies in real time](https://mlsanity.com/paper/2212.10249/) (2022)
- [Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks](https://mlsanity.com/paper/2306.00342/) (2023)

## Built on by (most cited first, 5 of 5)

- [Overcoming Rank Collapse in Feedback Alignment](https://mlsanity.com/paper/2606.11123/) (2026)
- [Pretraining with random noise for uncertainty calibration](https://mlsanity.com/paper/2412.17411/) (2024)
- [One-Time Soft Alignment Enables Resilient Learning without Weight Transport](https://mlsanity.com/paper/2505.20892/) (2025)
- [Learning What to Predict: Downstream-Guided Task Design for Continued Pretraining](https://mlsanity.com/paper/2601.22108/) (2026)
- [Target noise: A pre-training based neural network initialization for efficient high resolution learning](https://mlsanity.com/paper/2602.06585/) (2026)

## Abstract (arXiv)

The brain prepares for learning even before interacting with the environment, by refining and optimizing its structures through spontaneous neural activity that resembles random noise. However, the mechanism of such a process has yet to be thoroughly understood, and it is unclear whether this process can benefit the algorithm of machine learning. Here, we study this issue using a neural network with a feedback alignment algorithm, demonstrating that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport. First, we found that random noise training modifies forward weights to match backward synaptic feedback, which is necessary for teaching errors by feedback alignment. As a result, a network with pre-aligned weights learns notably faster than a network without random noise training, even reaching a convergence speed comparable to that of a backpropagation algorithm. Sequential training with both random noise and data brings weights closer to synaptic feedback than training solely with data, enabling more precise credit assignment and faster learning. We also found that each readout probability approaches the chance level and that the effective dimensionality of weights decreases in a network pretrained with random noise. This pre-regularization allows the network to learn simple solutions of a low rank, reducing the generalization loss during subsequent training. This also enables the network robustly to generalize a novel, out-of-distribution dataset. Lastly, we confirmed that random noise pretraining reduces the amount of meta-loss, enhancing the network ability to adapt to various tasks. Overall, our results suggest that random noise training with feedback alignment offers a straightforward yet effective method of pretraining that facilitates quick and reliable learning without weight transport.