mlsanityData to 7 Oct 2026

What's Hidden in a Randomly Weighted Neural Network?

arXiv 1911.13299 ↗First posted 29 Nov 2019cs.CV cs.LGcited by 236

Not on the map. Why?

Abstract (arXiv)

Training a neural network is synonymous with learning the values of the weights. By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an "untrained subnetwork" approaches a network with learned weights in accuracy. Our code and pretrained models are available at https://github.com/allenai/hidden-networks.

Key facts. Cited by 236 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 38; 2022: 62; 2023: 49; 2024: 39; 2025: 32; 2026: 16.

236 papers.

Sort:

By area:

  1. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  2. 2023VeRA: Vector-based Random Matrix Adaptationcited by 294
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  4. 2021Block Pruning For Faster Transformerscited by 176
  5. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  6. 2022A Theoretical Study on Solving Continual Learningcited by 78
  7. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  8. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  9. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  10. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  11. 2022Advancing Model Pruning via Bi-level Optimizationcited by 55
  12. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
38 more
  1. 2022Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?cited by 47
  2. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  3. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  4. 2021Recent Advances on Neural Network Pruning at Initializationcited by 41
  5. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  6. 2022Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?cited by 38
  7. 2022On the Soft-Subnetwork for Few-shot Class Incremental Learningcited by 37
  8. 2022Rare Gems: Finding Lottery Tickets at Initializationcited by 34
  9. 2021Learning where to learn: Gradient sparsity in meta and continual learningcited by 33
  10. 2022Sparse Random Networks for Communication-Efficient Federated Learningcited by 32
  11. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  12. 2021On the Existence of Universal Lottery Ticketscited by 30
  13. 2021Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networkscited by 29
  14. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  15. 2023Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspectivecited by 23
  16. 2024A Survey of Lottery Ticket Hypothesiscited by 23
  17. 2023Open-World Continual Learning: Unifying Novelty Detection and Continual Learningcited by 22
  18. 2022Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Networkcited by 21
  19. 2022Most Activation Functions Can Win the Lottery Without Excessive Depthcited by 21
  20. 2021Pruning Randomly Initialized Neural Networks with Iterative Randomizationcited by 20
  21. 2022Lifelong Reinforcement Learning with Modulating Maskscited by 20
  22. 2022DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networkscited by 19
  23. 2022A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesiscited by 19
  24. 2021Knowledge Evolution in Neural Networkscited by 18
  25. 2023One Network, Many Masks: Towards More Parameter-Efficient Transfer Learningcited by 18
  26. 2023Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitivescited by 18
  27. 2024How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teacherscited by 18
  28. 2024Algorithmic Capabilities of Random Transformerscited by 17
  29. 2022Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNscited by 16
  30. 2022Convolutional and Residual Networks Provably Contain Lottery Ticketscited by 16
  31. 2023Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruningcited by 16
  32. 2023Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capabilitycited by 16
  33. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 16
  34. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  35. 2024LEIA: Latent View-invariant Embeddings for Implicit 3D Articulationcited by 16
  36. 2024On the Surprising Effectiveness of Attention Transfer for Vision Transformerscited by 16
  37. 2021A Probabilistic Approach to Neural Network Pruningcited by 15
  38. 2024Masks, Signs, And Learning Rate Rewindingcited by 15

50 most cited of 236 All 236 (JSON)