mlsanityData to 7 Oct 2026

Finding Everything within Random Binary Networks

arXiv 2110.08996 ↗First posted 18 Oct 2021cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

A recent work by Ramanujan et al. (2020) provides significant empirical evidence that sufficiently overparameterized, random neural networks contain untrained subnetworks that achieve state-of-the-art accuracy on several predictive tasks. A follow-up line of theoretical work provides justification of these findings by proving that slightly overparameterized neural networks, with commonly used continuous-valued random initializations can indeed be pruned to approximate any target network. In this work, we show that the amplitude of those random weights does not even matter. We prove that any target network can be approximated up to arbitrary accuracy by simply pruning a random network of binary ±1 weights that is only a polylogarithmic factor wider and deeper than the target network.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 18 Oct 2021.

Builds on

27 of 39 references.

References · 27

  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  2. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  3. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  4. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  5. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  6. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  7. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  8. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  9. 2020What is the State of Neural Network Pruning?cited by 578
  10. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  11. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  12. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
15 more
  1. 2018AMC: AutoML for Model Compression and Acceleration on Mobile Devicescited by 375
  2. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 298
  3. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  4. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 212
  5. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  6. 2016Trained Ternary Quantizationcited by 203
  7. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  8. 2019Universal Approximation with Deep Narrow Networkscited by 198
  9. 2017Universal Function Approximation by Deep Neural Nets with Bounded Width and ReLU Activationscited by 104
  10. 2016Approximation by Combinations of ReLU and Squared ReLU Ridge Functions with ℓ1 and ℓ0 Controlscited by 84
  11. 2020Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficientcited by 81
  12. 2019Pruning from Scratchcited by 72
  13. 2020Logarithmic Pruning is All You Needcited by 71
  14. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  15. 2018The universal approximation power of finite-width deep ReLU networkscited by 11

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 1.

4 papers.

  1. 2022Rare Gems: Finding Lottery Tickets at Initializationcited by 34
  2. 2021The Augmented Image Prior: Distilling 1000 Classes by Extrapolating from a Single Imagecited by 7
  3. 2023Pruning Before Training May Improve Generalization, Provablycited by 3
  4. 2022"Understanding Robustness Lottery": A Geometric Visual Comparative Analysis of Neural Network Pruning Approachescited by 1