mlsanityData to 14 Oct 2026

HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization

arXiv 2308.07163 ↗First posted 14 Aug 2023cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Sparse neural networks are a key factor in developing resource-efficient machine learning applications. We propose the novel and powerful sparse learning method Adaptive Regularized Training (ART) to compress dense into sparse networks. Instead of the commonly used binary mask during training to reduce the number of model weights, we inherently shrink weights close to zero in an iterative manner with increasing weight regularization. Our method compresses the pre-trained model knowledge into the weights of highest magnitude. Therefore, we introduce a novel regularization loss named HyperSparse that exploits the highest weights while conserving the ability of weight exploration. Extensive experiments on CIFAR and TinyImageNet show that our method leads to notable performance gains compared to other sparsification methods, especially in extremely high sparsity regimes up to 99.8 percent model sparsity. Additional investigations provide new insights into the patterns that are encoded in weights with high magnitudes.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 38 of its 46 references that are in the map. First posted on arXiv 14 Aug 2023.

Builds on

38 of 46 references.

References · 38

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  8. 2017A Closer Look at Memorization in Deep Networkscited by 942
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  10. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  11. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  12. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
26 more
  1. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  2. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  3. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  4. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 416
  5. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  6. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 256
  7. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 183
  8. 2019What Do Compressed Deep Neural Networks Forget?cited by 171
  9. 2018Three Mechanisms of Weight Decay Regularizationcited by 113
  10. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  11. 2019Winning the Lottery with Continuous Sparsificationcited by 100
  12. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 96
  13. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  14. 2017Exploring Sparsity in Recurrent Neural Networkscited by 75
  15. 2020Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpotcited by 66
  16. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  17. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  18. 2022Training Your Sparse Neural Network Better with Any Maskcited by 26
  19. 2020Topological Insights into Sparse Neural Networkscited by 22
  20. 2022Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networkscited by 21
  21. 2022Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspectivecited by 9
  22. 2022Blind Knowledge Distillation for Robust Image Classificationcited by 9
  23. 2022Spartan: Differentiable Sparsity via Regularized Transportationcited by 8
  24. 2023Take 5: Interpretable Image Classification with a Handful of Featurescited by 7
  25. 2023Compensation Learning in Semantic Segmentationcited by 5
  26. 2022Pushing the Efficiency Limit Using Structured Sparse Convolutionscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 4; 2026: 4.

9 papers.

  1. 2025Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetriescited by 14
  2. 2023Q-SENN: Quantized Self-Explaining Neural Networkscited by 9
  3. 2025UncertainSAM: Fast and Efficient Uncertainty Quantification of the Segment Anything Modelcited by 4
  4. 2025Pruning by Block Benefit: Exploring the Properties of Vision Transformer Blocks during Domain Adaptationcited by 3
  5. 2025QPM: Discrete Optimization for Globally Interpretable Image Classificationcited by 2
  6. 2026Growing Networks with Autonomous Pruningcited by 0
  7. 2026Video Patch Pruning: Efficient Video Instance Segmentation via Early Token Reductioncited by 0
  8. 2026DINO-QPM: Adapting Visual Foundation Models for Globally Interpretable Image Classificationcited by 0
  9. 2026Adaptive Regularization for Sparsity Control in Bregman-Based Optimizerscited by 0