mlsanityData to 14 Oct 2026

Training Sparse Neural Networks

arXiv 1611.06694 ↗First posted 21 Nov 2016cs.CV cs.LGcited by 73

Not on the map. Why?

Abstract (arXiv)

Deep neural networks with lots of parameters are typically used for large-scale computer vision tasks such as image classification. This is a result of using dense matrix multiplications and convolutions. However, sparse computations are known to be much more efficient. In this work, we train and build neural networks which implicitly use sparse computations. We introduce additional gate variables to perform parameter selection and show that this is equivalent to using a spike-and-slab prior. We experimentally validate our method on both small and large networks and achieve state-of-the-art compression results for sparse neural network models.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 20; 2022: 15; 2023: 16; 2024: 12; 2025: 8; 2026: 2.

73 papers.

Sort:

By area:

  1. 2021CF-GNNExplainer: Counterfactual Explanations for Graph Neural Networkscited by 119
  2. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  3. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 84
  4. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  5. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  6. 2021Efficient Neural Network Training via Forward and Backward Propagation Sparsificationcited by 30
  7. 2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
  8. 2022A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challengescited by 30
  9. 2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
  10. 2021GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarizationcited by 29
  11. 2023AutoReP: Automatic ReLU Replacement for Fast Private Network Inferencecited by 29
  12. 2023LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inferencecited by 27
38 more
  1. 2021Effective Model Sparsification by Scheduled Grow-and-Prune Methodscited by 26
  2. 2022Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constantscited by 24
  3. 2021A Bregman Learning Framework for Sparse Neural Networkscited by 15
  4. 2022DiSparse: Disentangled Sparsification for Multitask Model Compressioncited by 14
  5. 2022Spatial Mixture-of-Expertscited by 14
  6. 2021On the Robustness of Monte Carlo Dropout Trained with Noisy Labelscited by 11
  7. 2024Circuit Compositions: Exploring Modular Structures in Transformer-Based Language Modelscited by 11
  8. 2021Lottery Jackpots Exist in Pre-trained Modelscited by 10
  9. 2022Learning Signal Temporal Logic through Neural Network for Interpretable Classificationcited by 10
  10. 2023Efficient Symbolic Policy Learning with Differentiable Symbolic Expressioncited by 10
  11. 2022Sequential Attention for Feature Selectioncited by 9
  12. 2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
  13. 2024TLINet: Differentiable Neural Network Temporal Logic Inferencecited by 9
  14. 2022Accelerating Attention through Gradient-Based Learned Runtime Pruningcited by 8
  15. 2023Balanced Training for Sparse GANscited by 8
  16. 2024Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradientcited by 6
  17. 2021Neural Network for Weighted Signal Temporal Logiccited by 5
  18. 2021Sparse Personalized Federated Learningcited by 5
  19. 2022Pruning-aware Sparse Regularization for Network Pruningcited by 5
  20. 2022Sparse Vicious Attacks on Graph Neural Networkscited by 5
  21. 2025Temporal Action Detection Model Compression by Progressive Block Dropcited by 5
  22. 2021[Reproducibility Report] Rigging the Lottery: Making All Tickets Winnerscited by 4
  23. 2022Sparse Federated Learning with Hierarchical Personalized Modelscited by 4
  24. 2022Interaction Decompositions for Tensor Network Regressioncited by 4
  25. 2022Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionalitycited by 4
  26. 2022Streaming, fast and accurate on-device Inverse Text Normalization for Automatic Speech Recognitioncited by 4
  27. 2023UPSCALE: Unconstrained Channel Pruningcited by 4
  28. 2024Deep Fr'echet Regressioncited by 4
  29. 2023DSFormer: Effective Compression of Text-Transformers by Dense-Sparse Weight Factorizationcited by 3
  30. 2024DIET: Customized Slimming for Incompatible Networks in Sequential Recommendationcited by 3
  31. 2025Boosting Large Language Models with Mask Fine-Tuningcited by 3
  32. 2021End-to-End Supermask Pruning: Learning to Prune Image Captioning Modelscited by 2
  33. 2023Effective Neural Network L0 Regularization With BinMaskcited by 2
  34. 2024Zero-Space Cost Fault Tolerance for Transformer-based Language Models on ReRAMcited by 2
  35. 2024Mini-batch Estimation for Deep Cox Models: Statistical Foundations and Practical Guidancecited by 2
  36. 2025Graph Inverse Style Transfer for Counterfactual Explainabilitycited by 2
  37. 2023Learning Activation Functions for Sparse Neural Networkscited by 1
  38. 2023Binary domain generalization for sparsifying binary neural networkscited by 1

50 most cited of 73 All 73 (JSON)