mlsanityData to 7 Oct 2026

Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training

arXiv 2102.02887 ↗First posted 4 Feb 2021cs.LG cs.AI cs.CVcited by 107

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a new perspective on training deep neural networks capable of state-of-the-art performance without the need for the expensive over-parameterization by proposing the concept of In-Time Over-Parameterization (ITOP) in sparse training. By starting from a random sparse network and continuously exploring sparse connectivities during training, we can perform an Over-Parameterization in the space-time manifold, closing the gap in the expressibility between sparse training and dense training. We further use ITOP to understand the underlying mechanism of Dynamic Sparse Training (DST) and indicate that the benefits of DST come from its ability to consider across time all possible parameters when searching for the optimal sparse connectivity. As long as there are sufficient parameters that have been reliably explored during training, DST can outperform the dense neural network by a large margin. We present a series of experiments to support our conjecture and achieve the state-of-the-art sparse training performance with ResNet-50 on ImageNet. More impressively, our method achieves dominant performance over the overparameterization-based sparse methods at extreme sparsity levels. When trained on CIFAR-100, our method can match the performance of the dense model even at an extreme sparsity (98%). Code can be found https://github.com/Shiweiliuiiiiiii/In-Time-Over-Parameterization.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 58 of its 74 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

58 of 74 references.

References · 58

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  11. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  12. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
38 more
  1. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  2. 2018Rethinking the Value of Network Pruningcited by 595
  3. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  4. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  5. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  6. 2019Importance Estimation for Neural Network Pruningcited by 531
  7. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  8. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  9. 2019The State of Sparsity in Deep Neural Networkscited by 471
  10. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  11. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  12. 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 313
  13. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  14. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  15. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  16. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  17. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  18. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  19. 2014Qualitatively characterizing neural network optimization problemscited by 186
  20. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  21. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  22. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  23. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  24. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  25. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  26. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  27. 2019Winning the Lottery with Continuous Sparsificationcited by 99
  28. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  29. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  30. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  31. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  32. 2016Training Sparse Neural Networkscited by 73
  33. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 66
  34. 2020Progressive Skeletonization: Trimming more fat from a network at initializationcited by 64
  35. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  36. 2017NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigmcited by 60
  37. 2018Compressing Neural Networks using the Variational Information Bottleneckcited by 59
  38. 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 30; 2023: 27; 2024: 23; 2025: 10; 2026: 7.

107 papers.

Sort:

By area:

  1. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  2. 2023Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsitycited by 163
  3. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  4. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  5. 2022The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Trainingcited by 85
  6. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  7. 2022DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Trainingcited by 74
  8. 2023Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMscited by 61
  9. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  10. 2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
  11. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  12. 2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
38 more
  1. 2024Kolmogorov-Arnold Network for Satellite Image Classification in Remote Sensingcited by 46
  2. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  3. 2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
  4. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  5. 2022NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networkscited by 38
  6. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  7. 2023Dynamic Sparse Training with Structured Sparsitycited by 36
  8. 2022Achieving Personalized Federated Learning with Sparse Local Modelscited by 30
  9. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  10. 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
  11. 2022Training Your Sparse Neural Network Better with Any Maskcited by 26
  12. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  13. 2021The Elastic Lottery Ticket Hypothesiscited by 23
  14. 2023LEMON: Lossless model expansioncited by 23
  15. 2022Dynamic Sparse Network for Time Series Classification: Learning What to "see''cited by 22
  16. 2026Flow caching for autoregressive video generationcited by 19
  17. 2022Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural Networkscited by 18
  18. 2022Towards Sparsification of Graph Neural Networkscited by 18
  19. 2023Dynamic Sparsity Is Channel-Level Sparsity Learnercited by 17
  20. 2022Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNscited by 16
  21. 2023Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Trainingcited by 16
  22. 2023E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentationcited by 16
  23. 2024Sparse maximal update parameterization: A holistic approach to sparse training dynamicscited by 16
  24. 2022Where to Pay Attention in Sparse Training for Feature Selection?cited by 15
  25. 2022Dynamic Sparse Training via Balancing the Exploration-Exploitation Trade-offcited by 15
  26. 2025Sculpting Memory: Multi-Concept Forgetting in Diffusion Models via Dynamic Mask and Concept-Aware Optimizationcited by 15
  27. 2021Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsitycited by 14
  28. 2024HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learningcited by 14
  29. 2023Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learningcited by 13
  30. 2023Junk DNA Hypothesis: Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMscited by 13
  31. 2021Towards Structured Dynamic Sparse Pre-Training of BERTcited by 12
  32. 2022Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Surveycited by 12
  33. 2023A Unified Framework for Soft Threshold Pruningcited by 12
  34. 2023Winning Prize Comes from Losing Tickets: Improve Invariant Learning by Exploring Variant Parameters for Out-of-Distribution Generalizationcited by 12
  35. 2025AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMscited by 12
  36. 2022Layer Freezing & Data Sieving: Missing Pieces of a Generic Framework for Sparse Trainingcited by 11
  37. 2024Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustnesscited by 11
  38. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimizationcited by 10

50 most cited of 107 All 107 (JSON)