# Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing

arXiv: https://arxiv.org/abs/1802.00844 · HTML: https://mlsanity.com/paper/1802.00844/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2018.

## Built on by (most cited first, 17 of 17)

- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [Gradient Routing: Masking Gradients to Localize Computation in Neural Networks](https://mlsanity.com/paper/2410.04332/) (2024)
- [Bootstrap Your Own Correspondences](https://mlsanity.com/paper/2106.00677/) (2021)
- [Improving Generalization of Adversarial Training via Robust Critical Fine-Tuning](https://mlsanity.com/paper/2308.02533/) (2023)
- [RPMixer: Shaking Up Time Series Forecasting with Random Projections for Large Spatial-Temporal Data](https://mlsanity.com/paper/2402.10487/) (2024)
- [Extend Your Own Correspondences: Unsupervised Distant Point Cloud Registration by Progressive Distance Extension](https://mlsanity.com/paper/2403.03532/) (2024)
- [Efficient and Private Federated Learning with Partially Trainable Networks](https://mlsanity.com/paper/2110.03450/) (2021)
- [Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey](https://mlsanity.com/paper/2205.08099/) (2022)
- [Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks](https://mlsanity.com/paper/2101.06475/) (2021)
- [Accelerating Multi-Objective Neural Architecture Search by Random-Weight Evaluation](https://mlsanity.com/paper/2110.05242/) (2021)
- [Expressivity of Neural Networks with Random Weights and Learned Biases](https://mlsanity.com/paper/2407.00957/) (2024)
- [Dolph2Vec: Self-Supervised Representations of Dolphin Vocalizations](https://mlsanity.com/paper/2606.12503/) (2026)
- [Exploring Heterogeneous Characteristics of Layers in ASR Models for More Efficient Training](https://mlsanity.com/paper/2110.04267/) (2021)
- [Simplifying Multi-Task Architectures Through Task-Specific Normalization](https://mlsanity.com/paper/2512.20420/) (2025)
- [ExplainFix: Explainable Spatially Fixed Deep Networks](https://mlsanity.com/paper/2303.10408/) (2023)
- [An Explainable Fast Deep Neural Network for Emotion Recognition](https://mlsanity.com/paper/2407.14865/) (2024)
- [Green-ELM: Efficient Analytic Learning via High-Dimensional Random Projections](https://mlsanity.com/paper/2604.15613/) (2026)

## Abstract (arXiv)

Training deep neural networks results in strong learned representations that show good generalization capabilities. In most cases, training involves iterative modification of all weights inside the network via back-propagation. In Extreme Learning Machines, it has been suggested to set the first layer of a network to fixed random values instead of learning it. In this paper, we propose to take this approach a step further and fix almost all layers of a deep convolutional neural network, allowing only a small portion of the weights to be learned. As our experiments show, fixing even the majority of the parameters of the network often results in performance which is on par with the performance of learning all of them. The implications of this intriguing property of deep neural networks are discussed and we suggest ways to harness it to create more robust representations.