Are Sparse Neural Networks Better Hard Sample Learners?
Not on the map. Why?
Abstract (arXiv)
While deep learning has demonstrated impressive progress, it remains a daunting challenge to learn from hard samples as these samples are usually noisy and intricate. These hard samples play a crucial role in the optimal performance of deep neural networks. Most research on Sparse Neural Networks (SNNs) has focused on standard training data, leaving gaps in understanding their effectiveness on complex and challenging data. This paper's extensive investigation across scenarios reveals that most SNNs trained on challenging samples can often match or surpass dense models in accuracy at certain sparsity levels, especially with limited data. We observe that layer-wise density ratios tend to play an important role in SNN performance, particularly for methods that train from scratch without pre-trained initialization. These insights enhance our understanding of SNNs' behavior and potential for efficient learning approaches in data-centric AI. Our code is publicly available at: urlhttps://github.com/QiaoXiao7282/hardsamplelearners.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 13 Sep 2024.
Builds on
52 of 67 references.
References · 52
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
- 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
- 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 669
- 2020Contrastive Learning with Hard Negative Samplescited by 619
- 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
- 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 529
- 2020Overfitting in adversarially robust deep learningcited by 504
38 more
- 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
- 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
- 2019The State of Sparsity in Deep Neural Networkscited by 471
- 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
- 2018Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Surveycited by 459
- 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
- 2023Scaling Data-Constrained Language Modelscited by 391
- 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
- 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
- 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
- 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
- 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
- 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 233
- 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
- 2021Data Augmentation Can Improve Robustnesscited by 193
- 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
- 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
- 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
- 2021Deep Learning Through the Lens of Example Difficultycited by 139
- 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
- 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
- 2020Estimating Example Difficulty Using Variance of Gradientscited by 96
- 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
- 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
- 2023Large-scale Dataset Pruning with Dynamic Uncertaintycited by 63
- 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
- 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48
- 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
- 2016A topological insight into restricted Boltzmann machinescited by 32
- 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
- 2021Low Curvature Activations Reduce Overfitting in Adversarial Trainingcited by 29
- 2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
- 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
- 2022Dynamic Sparse Network for Time Series Classification: Learning What to "see''cited by 22
- 2023E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentationcited by 16
- 2021Truly Sparse Neural Networks at Scalecited by 15
- 2022Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficultycited by 5
- 2024Dynamic Data Pruning for Automatic Speech Recognitioncited by 4
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.