Effective Neural Network L0 Regularization With BinMask
Not on the map. Why?
Abstract (arXiv)
L0 regularization of neural networks is a fundamental problem. In addition to regularizing models for better generalizability, L0 regularization also applies to selecting input features and training sparse neural networks. There is a large body of research on related topics, some with quite complicated methods. In this paper, we show that a straightforward formulation, BinMask, which multiplies weights with deterministic binary masks and uses the identity straight-through estimator for backpropagation, is an effective L0 regularizer. We evaluate BinMask on three tasks: feature selection, network sparsification, and model regularization. Despite its simplicity, BinMask achieves competitive performance on all the benchmarks without task-specific tuning compared to methods designed for each task. Our results suggest that decoupling weights from mask optimization, which has been widely adopted by previous work, is a key component for effective L0 regularization.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 31 of its 53 references that are in the map. First posted on arXiv 21 Apr 2023.
Builds on
31 of 53 references.
References · 31
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
- 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
- 2020What is the State of Neural Network Pruning?cited by 583
- 2014OpenML: networked science in machine learningcited by 526
- 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 521
19 more
- 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
- 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
- 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 182
- 2019LassoNet: A Neural Network with Feature Sparsitycited by 82
- 2016Training Sparse Neural Networkscited by 73
- 2015A Review of Feature Selection Methods Based on Mutual Informationcited by 67
- 2019The Difficulty of Training Sparse Neural Networkscited by 62
- 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
- 2018DeepPINK: reproducible feature selection in deep neural networkscited by 41
- 2020Efficient Exact Verification of Binarized Neural Networkscited by 33
- 2020Feature Importance Ranking for Deep Learningcited by 31
- 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
- 2020The Two Regimes of Deep Network Trainingcited by 30
- 2020Quick and Robust Feature Selection: the Strength of Energy-efficient Sparse Training for Autoencoderscited by 28
- 2019Back to Simplicity: How to Train Accurate BNNs from Scratch?cited by 24
- 2020Training Binary Neural Networks using the Bayesian Learning Rulecited by 23
- 2020Reintroducing Straight-Through Estimators as Principled Methods for Stochastic Binary Networkscited by 15
- 2022Are Straight-Through gradients and Soft-Thresholding all you need for Sparse Training?cited by 9
- 2023Supervised Feature Selection with Neuron Evolution in Sparse Neural Networkscited by 7
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.