mlsanityData to 7 Oct 2026

The Difficulty of Training Sparse Neural Networks

arXiv 1906.10732 ↗First posted 25 Jun 2019cs.LG cs.CV stat.MLcited by 62

Not on the map. Why?

Abstract (arXiv)

We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime. Recent work of citepGale2019, Liu2018 has shown that sparse ResNet-50 architectures trained on ImageNet-2012 dataset converge to solutions that are significantly worse than those found by pruning. We show that, despite the failure of optimizers, there is a linear path with a monotonically decreasing objective from the initialization to the "good" solution. Additionally, our attempts to find a decreasing objective path from "bad" solutions to the "good" ones in the sparse subspace fail. However, if we allow the path to traverse the dense subspace, then we consistently find a path between two solutions. These findings suggest traversing extra dimensions may be needed to escape stationary points found in the sparse subspace.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 13; 2023: 13; 2024: 14; 2025: 3; 2026: 2.

62 papers.

Sort:

  1. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  2. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 177
  3. 2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
  4. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 107
  5. 2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
  6. 2021Learning Neural Network Subspacescited by 82
  7. 2021Top-KAST: Top-K Always Sparse Trainingcited by 79
  8. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  9. 2021Powerpropagation: A sparsity inducing weight reparameterisationcited by 49
  10. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  11. 2021Efficient Lottery Ticket Finding: Less Data is Morecited by 44
  12. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
38 more
  1. 2022Sparsity Winning Twice: Better Robust Generalization from More Efficient Trainingcited by 41
  2. 2025Weight-sparse transformers have interpretable circuitscited by 40
  3. 2021GANs Can Play Lottery Tickets Toocited by 32
  4. 2023Towards provably efficient quantum algorithms for large-scale machine-learning modelscited by 32
  5. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  6. 2021Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networkscited by 29
  7. 2022Sparse Double Descent: Where Network Pruning Aggravates Overfittingcited by 26
  8. 2022Training Your Sparse Neural Network Better with Any Maskcited by 26
  9. 2023SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Modelscited by 26
  10. 2021Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspectivecited by 25
  11. 2023Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researcherscited by 25
  12. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  13. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 17
  14. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  15. 2023A Three-regime Model of Network Pruningcited by 15
  16. 2022To update or not to update? Neurons at equilibrium in deep modelscited by 13
  17. 2024Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustnesscited by 11
  18. 2022Training Recipe for N:M Structured Sparsity with Decaying Pruning Maskcited by 10
  19. 2023Accurate Neural Network Pruning Requires Rethinking Sparse Optimizationcited by 10
  20. 2024Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Frameworkcited by 10
  21. 2024No Free Prune: Information-Theoretic Barriers to Pruning at Initializationcited by 8
  22. 2021S3: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networkscited by 7
  23. 2022Don't Be So Dense: Sparse-to-Sparse GAN Training Without Sacrificing Performancecited by 7
  24. 2022Superposing Many Tickets into One: A Performance Booster for Sparse Neural Network Trainingcited by 6
  25. 2025Eau De Q-Network: Adaptive Distillation of Neural Networks in Deep Reinforcement Learningcited by 6
  26. 2021Estimation of a regression function on a manifold by fully connected deep neural networkscited by 5
  27. 2022Data-Efficient Double-Win Lottery Tickets from Robust Pre-trainingcited by 5
  28. 2024Theoretical characterisation of the Gauss-Newton conditioning in Neural Networkscited by 5
  29. 2023Calibrating the Rigged Lottery: Making All Tickets Reliablecited by 4
  30. 2024Scaling Laws for Post Training Quantized Large Language Modelscited by 4
  31. 2021Search Spaces for Neural Model Trainingcited by 3
  32. 2026GradPruner: Gradient-Guided Layer Pruning Enabling Efficient Fine-Tuning and Inference for LLMscited by 3
  33. 2022A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networkscited by 2
  34. 2023Vision Models Can Be Efficiently Specialized via Few-Shot Task-Aware Compressioncited by 2
  35. 2023Effective Neural Network L0 Regularization With BinMaskcited by 2
  36. 2023Understanding Sparse Neural Networks from their Topology via Multipartite Graph Representationscited by 2
  37. 2024Causal inference through multi-stage learning and doubly robust deep neural networkscited by 2
  38. 2024Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMscited by 2

50 most cited of 62 All 62 (JSON)