mlsanityData to 7 Oct 2026

Neural Network Reduction with Guided Regularizers

arXiv 2305.18448 ↗First posted 29 May 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Regularization techniques such as L1 and L2 regularizers are effective in sparsifying neural networks (NNs). However, to remove a certain neuron or channel in NNs, all weight elements related to that neuron or channel need to be prunable, which is not guaranteed by traditional regularization. This paper proposes a simple new approach named "Guided Regularization" that prioritizes the weights of certain NN units more than others during training, which renders some of the units less important and thus, prunable. This is different from the scattered sparsification of L1 and L2 regularizers where the the components of a weight matrix that are zeroed out can be located anywhere. The proposed approach offers a natural reduction of NN in the sense that a model is being trained while also neutralizing unnecessary units. We empirically demonstrate that our proposed method is effective in pruning NNs while maintaining performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 36 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

25 of 36 references.

References · 25

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  8. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  9. 2020What is the State of Neural Network Pruning?cited by 578
  10. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  12. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
13 more
  1. 2013Predicting Parameters in Deep Learningcited by 223
  2. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  3. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 189
  4. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  5. 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
  6. 2020Good Subnetworks Provably Exist: Pruning via Greedy Forward Selectioncited by 70
  7. 2019Data-Independent Neural Pruning via Coresetscited by 35
  8. 2015Diversity Networks: Neural Network Compression Using Determinantal Point Processescited by 25
  9. 2022Data-Efficient Structured Pruning via Submodular Optimizationcited by 22
  10. 2014Memory Bounded Deep Convolutional Networkscited by 17
  11. 2020Data-Independent Structured Pruning of Neural Networks via Coresetscited by 17
  12. 2019Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networkscited by 12
  13. 2018Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compressioncited by 9

Built on by

Not cited yet.