mlsanityData to 14 Oct 2026

Learning Sparse Networks Using Targeted Dropout

arXiv 1905.13678 ↗First posted 31 May 2019cs.LG stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

Neural networks are easier to optimise when they have many more weights than are required for modelling the mapping from inputs to outputs. This suggests a two-stage learning procedure that first learns a large net and then prunes away connections or hidden units. But standard training does not necessarily encourage nets to be amenable to pruning. We introduce targeted dropout, a method for training a neural network so that it is robust to subsequent pruning. Before computing the gradients for each weight update, targeted dropout stochastically selects a set of units or weights to be dropped using a simple self-reinforcing sparsity criterion and then computes the gradients for the remaining weights. The resulting network is robust to post hoc pruning of weights or units that frequently occur in the dropped sets. The method improves upon more complicated sparsifying regularisers while being simple to implement and easy to tune.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). First posted on arXiv 31 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 9; 2023: 11; 2024: 5; 2025: 3; 2026: 1.

40 papers.

Sort:

By area:

  1. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  2. 2021DICE: Leveraging Sparsification for Out-of-Distribution Detectioncited by 166
  3. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  4. 2021Differentiable Subset Pruning of Transformer Headscited by 47
  5. 2021Boosting Co-teaching with Compression Regularization for Label Noisecited by 31
  6. 2024STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruningcited by 30
  7. 2021Consistent Sparse Deep Learning: Theory and Computationcited by 27
  8. 2023Dropout Reduces Underfittingcited by 21
  9. 2021Layer-wise Model Pruning based on Mutual Informationcited by 19
  10. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17
  11. 2023How to Overcome Curse-of-Dimensionality for Out-of-Distribution Detection?cited by 15
  12. 2022To update or not to update? Neurons at equilibrium in deep modelscited by 13
28 more
  1. 2021SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networkscited by 12
  2. 2023Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inferencecited by 9
  3. 2025PocketSR: The Super-Resolution Expert in Your Pocket Mobilescited by 8
  4. 2022Compressing Features for Learning with Noisy Labelscited by 7
  5. 2022A Survey on Dropout Methods and Experimental Verification in Recommendationcited by 4
  6. 2022Controlled Dropout for Uncertainty Estimationcited by 4
  7. 2024Optimal Parameter and Neuron Pruning for Out-of-Distribution Detectioncited by 4
  8. 2021The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networkscited by 3
  9. 2022Regularization-based Pruning of Irrelevant Weights in Deep Neural Architecturescited by 3
  10. 2023Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selectioncited by 3
  11. 2021Learning by Active Forgetting for Neural Networkscited by 2
  12. 2022Structural Dropout for Model Width Compressioncited by 2
  13. 2023Dropout Drops Double Descentcited by 2
  14. 2023Dropout Attackscited by 2
  15. 2025Regularization can make diffusion models more efficientcited by 2
  16. 2025Deep Neural Networks Inspired by Differential Equationscited by 2
  17. 2023Learning Prescriptive ReLU Networkscited by 1
  18. 2023Binary domain generalization for sparsifying binary neural networkscited by 1
  19. 2023Detecting and Learning Out-of-Distribution Data in the Open world: Algorithm and Theorycited by 1
  20. 2024AdaFSNet: Time Series Classification Based on Convolutional Network with a Adaptive and Effective Kernel Size Configurationcited by 1
  21. 2021ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spacescited by 0
  22. 2021Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Searchcited by 0
  23. 2022Neural Network Optimization for Reinforcement Learning Tasks Using Sparse Computationscited by 0
  24. 2022Sparsifying Binary Networkscited by 0
  25. 2023Revisiting Implicit Models: Sparsity Trade-offs Capability in Weight-tied Model for Vision Taskscited by 0
  26. 2024Multi-level projection with exponential parallel speedup; Application to sparse auto-encoders neural networkscited by 0
  27. 2024A new Linear Time Bi-level ℓ1,∞ projection ; Application to the sparsification of auto-encoders neural networkscited by 0
  28. 2026Regularisation in neural networks: a survey and empirical analysis of approachescited by 0