mlsanityData to 7 Oct 2026

Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness

arXiv 2410.03030 ↗First posted 3 Oct 2024cs.CV cs.AIcited by 11

Not on the map. Why?

Abstract (arXiv)

It is generally perceived that Dynamic Sparse Training opens the door to a new era of scalability and efficiency for artificial neural networks at, perhaps, some costs in accuracy performance for the classification task. At the same time, Dense Training is widely accepted as being the "de facto" approach to train artificial neural networks if one would like to maximize their robustness against image corruption. In this paper, we question this general practice. Consequently, we claim that, contrary to what is commonly thought, the Dynamic Sparse Training methods can consistently outperform Dense Training in terms of robustness accuracy, particularly if the efficiency aspect is not considered as a main objective (i.e., sparsity levels between 10% and up to 50%), without adding (or even reducing) resource cost. We validate our claim on two types of data, images and videos, using several traditional and modern deep learning architectures for computer vision and three widely studied Dynamic Sparse Training algorithms. Our findings reveal a new yet-unknown benefit of Dynamic Sparse Training and open new possibilities in improving deep learning robustness beyond the current state of the art.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 61 of its 77 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

61 of 77 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  12. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
38 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  2. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  3. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  4. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  5. 2019A Fourier Perspective on Model Robustness in Computer Visioncited by 288
  6. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  7. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  8. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  9. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  10. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  11. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  12. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  13. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  14. 20223D Common Corruptions and Data Augmentationcited by 120
  15. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  16. 2019Image Classification with Deep Learning in the Presence of Noisy Labels: A Surveycited by 101
  17. 2021On Interaction Between Augmentations and Corruptions in Natural Corruption Robustnesscited by 98
  18. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 89
  19. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  20. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  21. 2020Gradient Flow in Sparse Neural Networks and How Lottery Tickets Wincited by 77
  22. 2021MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edgecited by 77
  23. 2019The Difficulty of Training Sparse Neural Networkscited by 62
  24. 2021Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracycited by 62
  25. 2021A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustnesscited by 62
  26. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  27. 2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
  28. 2019Sparse evolutionary Deep Learning with over one million artificial neurons on commodity hardwarecited by 48
  29. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  30. 2021PRIME: A few primitives can boost robustness to common corruptionscited by 40
  31. 2021Improving robustness against common corruptions with frequency biased modelscited by 34
  32. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  33. 2021WaveCNet: Wavelet Integrated CNNs to Suppress Aliasing Effect for Noise-Robust Image Classificationcited by 29
  34. 2022Large-scale Robustness Analysis of Video Action Recognition Modelscited by 29
  35. 2022FrequencyLowCut Pooling -- Plug & Play against Catastrophic Overfittingcited by 28
  36. 2020Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoderscited by 27
  37. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  38. 2022Dynamic Sparse Network for Time Series Classification: Learning What to "see''cited by 22

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 6.

11 papers.

  1. 2025Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolutioncited by 1
  2. 2024Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruningcited by 0
  3. 2025NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensemblingcited by 0
  4. 2025Addressing the Collaboration Dilemma in Low-Data Federated Learning via Transient Sparsitycited by 0
  5. 2025A Recovery Guarantee for Sparse Neural Networkscited by 0
  6. 2026Topology-Aware Revival for Efficient Sparse Trainingcited by 0
  7. 2026SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Trainingcited by 0
  8. 2026Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scalingcited by 0
  9. 2026Learning Subset-Shared Invariances for Domain Generalization with Mixture-of-Expertscited by 0
  10. 2026Robustness Emerges Early in Training Dynamics, but Is Not Preservedcited by 0
  11. 2026Suppress and Diversify: Refining Robust Pathways for Corruption Robustnesscited by 0