# What's Hidden in a Randomly Weighted Neural Network?

arXiv: https://arxiv.org/abs/1911.13299 · HTML: https://mlsanity.com/paper/1911.13299/

## Key facts

- Cited by 237 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2019.

## Built on by (most cited first, 30 of 237)

- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [VeRA: Vector-based Random Matrix Adaptation](https://mlsanity.com/paper/2310.11454/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [Asymmetry in Low-Rank Adapters of Foundation Models](https://mlsanity.com/paper/2402.16842/) (2024)
- [A Theoretical Study on Solving Continual Learning](https://mlsanity.com/paper/2211.02633/) (2022)
- [Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA](https://mlsanity.com/paper/2410.20672/) (2024)
- [A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness](https://mlsanity.com/paper/2106.09129/) (2021)
- [Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs](https://mlsanity.com/paper/2310.08915/) (2023)
- [Advancing Model Pruning via Bi-level Optimization](https://mlsanity.com/paper/2210.04092/) (2022)
- [Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network](https://mlsanity.com/paper/2103.09377/) (2021)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?](https://mlsanity.com/paper/2203.08392/) (2022)
- [Effective Sparsification of Neural Networks with Global Sparsity Constraint](https://mlsanity.com/paper/2105.01571/) (2021)
- [Break It Down: Evidence for Structural Compositionality in Neural Networks](https://mlsanity.com/paper/2301.10884/) (2023)
- [Recent Advances on Neural Network Pruning at Initialization](https://mlsanity.com/paper/2103.06460/) (2021)
- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?](https://mlsanity.com/paper/2210.03044/) (2022)
- [On the Soft-Subnetwork for Few-shot Class Incremental Learning](https://mlsanity.com/paper/2209.07529/) (2022)
- [Rare Gems: Finding Lottery Tickets at Initialization](https://mlsanity.com/paper/2202.12002/) (2022)
- [Learning where to learn: Gradient sparsity in meta and continual learning](https://mlsanity.com/paper/2110.14402/) (2021)
- [Sparse Random Networks for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2209.15328/) (2022)
- [Efficient Neural Network Training via Forward and Backward Propagation Sparsification](https://mlsanity.com/paper/2111.05685/) (2021)
- [On the Existence of Universal Lottery Tickets](https://mlsanity.com/paper/2111.11146/) (2021)
- [Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks](https://mlsanity.com/paper/2110.14068/) (2021)
- [Why Random Pruning Is All We Need to Start Sparse](https://mlsanity.com/paper/2210.02412/) (2022)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective](https://mlsanity.com/paper/2312.01397/) (2023)
- [Lifelong Reinforcement Learning with Modulating Masks](https://mlsanity.com/paper/2212.11110/) (2022)
- [Open-World Continual Learning: Unifying Novelty Detection and Continual Learning](https://mlsanity.com/paper/2304.10038/) (2023)

## Abstract (arXiv)

Training a neural network is synonymous with learning the values of the weights. By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an "untrained subnetwork" approaches a network with learned weights in accuracy. Our code and pretrained models are available at https://github.com/allenai/hidden-networks.