mlsanityData to 14 Oct 2026

Pruning Randomly Initialized Neural Networks with Iterative Randomization

arXiv 2106.09269 ↗First posted 17 Jun 2021cs.LG cs.AI cs.NEcited by 20

Not on the map. Why?

Abstract (arXiv)

Pruning the weights of randomly initialized neural networks plays an important role in the context of lottery ticket hypothesis. Ramanujan et al. (2020) empirically showed that only pruning the weights can achieve remarkable performance instead of optimizing the weight values. However, to achieve the same level of performance as the weight optimization, the pruning approach requires more parameters in the networks before pruning and thus more memory space. To overcome this parameter inefficiency, we introduce a novel framework to prune randomly initialized neural networks with iteratively randomizing weight values (IteRand). Theoretically, we prove an approximation theorem in our framework, which indicates that the randomizing operations are provably effective to reduce the required number of the parameters. We also empirically demonstrate the parameter efficiency in multiple experiments on CIFAR-10 and ImageNet. The code is available at: https://github.com/dchiji-ntt/iterand

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 17 Jun 2021.

Builds on

25 of 38 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2016Wide Residual Networkscited by 3,070
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  9. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  10. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  11. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  12. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
13 more
  1. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  2. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  3. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  4. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 237
  5. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 207
  6. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 201
  7. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 189
  8. 2017Deep Rewiring: Training very sparse deep networkscited by 139
  9. 2019One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizerscited by 102
  10. 2020Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficientcited by 81
  11. 2019Pruning from Scratchcited by 72
  12. 2020Logarithmic Pruning is All You Needcited by 71
  13. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 7; 2024: 1; 2025: 3; 2026: 1.

20 papers.

Sort:

  1. 2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
  2. 2024Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Modelscited by 40
  3. 2022You Can Have Better Graph Neural Networks by Not Training Weights at All: Finding Untrained GNNs Ticketscited by 14
  4. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
  5. 2023Differentiable Transportation Pruningcited by 10
  6. 2023Sparse Binary Transformers for Multivariate Time Series Modelingcited by 8
  7. 2022Signing the Supermask: Keep, Hide, Invertcited by 7
  8. 2022Robust Binary Models by Pruning Randomly-initialized Networkscited by 5
  9. 2025Are Spatial-Temporal Graph Convolution Networks for Human Action Recognition Over-Parameterized?cited by 5
  10. 2022Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networkscited by 4
  11. 2022Strong Lottery Ticket Hypothesis with varepsilon--perturbationcited by 3
  12. 2023Randomly Initialized Subnetworks with Iterative Weight Recyclingcited by 3
8 more
  1. 2023Benchmarking Adversarial Robustness of Compressed Deep Learning Modelscited by 3
  2. 2023Pruning at Initialization -- A Sketching Perspectivecited by 2
  3. 2025The Strong Lottery Ticket Hypothesis for Multi-Head Attention Mechanismscited by 2
  4. 2026D2Prune: Sparsifying Large Language Models via Dual Taylor Expansion and Attention Distribution Awarenesscited by 2
  5. 2023Considering Layerwise Importance in the Lottery Ticket Hypothesiscited by 1
  6. 2023On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guaranteecited by 1
  7. 2022CWP: Instance complexity weighted channel-wise soft masks for network pruningcited by 0
  8. 2025The Quest for Winning Tickets in Low-Rank Adapterscited by 0