# Learning Sparse Neural Networks via Sensitivity-Driven Regularization

arXiv: https://arxiv.org/abs/1810.11764 · HTML: https://mlsanity.com/paper/1810.11764/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2018.

## Built on by (most cited first, 26 of 26)

- [Compressing Explicit Voxel Grid Representations: fast NeRFs become also small](https://mlsanity.com/paper/2210.12782/) (2022)
- [RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch](https://mlsanity.com/paper/2205.15043/) (2022)
- [Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation](https://mlsanity.com/paper/2104.11939/) (2021)
- [To update or not to update? Neurons at equilibrium in deep models](https://mlsanity.com/paper/2207.09455/) (2022)
- [SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks](https://mlsanity.com/paper/2102.03773/) (2021)
- [Is Oracle Pruning the True Oracle?](https://mlsanity.com/paper/2412.00143/) (2024)
- [Mini but Mighty: Finetuning ViTs with Mini Adapters](https://mlsanity.com/paper/2311.03873/) (2023)
- [Pruning-aware Sparse Regularization for Network Pruning](https://mlsanity.com/paper/2201.06776/) (2022)
- [HEMP: High-order Entropy Minimization for neural network comPression](https://mlsanity.com/paper/2107.05298/) (2021)
- [Automatic Sparse Connectivity Learning for Neural Networks](https://mlsanity.com/paper/2201.05020/) (2022)
- [Amenable Sparse Network Investigator](https://mlsanity.com/paper/2202.09284/) (2022)
- [Regularization-based Pruning of Irrelevant Weights in Deep Neural Architectures](https://mlsanity.com/paper/2204.04977/) (2022)
- [Towards Efficient Capsule Networks](https://mlsanity.com/paper/2208.09203/) (2022)
- [Learning sparse auto-encoders for green AI image coding](https://mlsanity.com/paper/2209.04448/) (2022)
- [Capsule Networks Do Not Need to Model Everything](https://mlsanity.com/paper/2204.01298/) (2022)
- [Engram Memory Encoding and Retrieval: A Neurocomputational Perspective](https://mlsanity.com/paper/2506.01659/) (2025)
- [Sparse Super-Regular Networks](https://mlsanity.com/paper/2201.01363/) (2022)
- [Improve Convolutional Neural Network Pruning by Maximizing Filter Variety](https://mlsanity.com/paper/2203.05807/) (2022)
- [VFDS: Variational Foresight Dynamic Selection in Bayesian Neural Networks for Efficient Human Activity Recognition](https://mlsanity.com/paper/2204.00130/) (2022)
- [Near-Linear Time Projection onto the ℓ1,∞ Ball; Application to Sparse Autoencoders](https://mlsanity.com/paper/2307.09836/) (2023)
- [Efficient Adaptation of Deep Neural Networks for Semantic Segmentation in Space Applications](https://mlsanity.com/paper/2504.15991/) (2025)
- [MinConvNets: A new class of multiplication-less Neural Networks](https://mlsanity.com/paper/2101.09492/) (2021)
- [Training Sparse Neural Network by Constraining Synaptic Weight on Unit Lp Sphere](https://mlsanity.com/paper/2103.16013/) (2021)
- [Description of Structural Biases and Associated Data in Sensor-Rich Environments](https://mlsanity.com/paper/2104.04885/) (2021)
- [Combining Past, Present and Future: A Self-Supervised Approach for Class Incremental Learning](https://mlsanity.com/paper/2311.08764/) (2023)
- [Multi-level projection with exponential parallel speedup; Application to sparse auto-encoders neural networks](https://mlsanity.com/paper/2405.02086/) (2024)

## Abstract (arXiv)

The ever-increasing number of parameters in deep neural networks poses challenges for memory-limited applications. Regularize-and-prune methods aim at meeting these challenges by sparsifying the network weights. In this context we quantify the output sensitivity to the parameters (i.e. their relevance to the network output) and introduce a regularization term that gradually lowers the absolute value of parameters with low sensitivity. Thus, a very large fraction of the parameters approach zero and are eventually set to zero by simple thresholding. Our method surpasses most of the recent techniques both in terms of sparsity and error rates. In some cases, the method reaches twice the sparsity obtained by other techniques at equal error rates.