# SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks

arXiv: https://arxiv.org/abs/2102.03773 · HTML: https://mlsanity.com/paper/2102.03773/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 7 Feb 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Empirical Analysis of the Hessian of Over-Parametrized Neural Networks](https://mlsanity.com/paper/1706.04454/) (2017)
- [Variational Dropout Sparsifies Deep Neural Networks](https://mlsanity.com/paper/1701.05369/) (2017)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [Soft Weight-Sharing for Neural Network Compression](https://mlsanity.com/paper/1702.04008/) (2017)
- [Channel Pruning via Automatic Structure Search](https://mlsanity.com/paper/2001.08565/) (2020)
- [Fast ConvNets Using Group-wise Brain Damage](https://mlsanity.com/paper/1506.02515/) (2015)
- [Pruning from Scratch](https://mlsanity.com/paper/1909.12579/) (2019)
- [Learning Sparse Networks Using Targeted Dropout](https://mlsanity.com/paper/1905.13678/) (2019)
- [Learning Sparse Neural Networks via Sensitivity-Driven Regularization](https://mlsanity.com/paper/1810.11764/) (2018)
- [Pruning artificial neural networks: a way to find well-generalizing, high-entropy sharp minima](https://mlsanity.com/paper/2004.14765/) (2020)
- [Post-synaptic potential regularization has potential](https://mlsanity.com/paper/1907.08544/) (2019)

## Built on by (most cited first, 12 of 12)

- [Compressing Explicit Voxel Grid Representations: fast NeRFs become also small](https://mlsanity.com/paper/2210.12782/) (2022)
- [To update or not to update? Neurons at equilibrium in deep models](https://mlsanity.com/paper/2207.09455/) (2022)
- [The rise of the lottery heroes: why zero-shot pruning is hard](https://mlsanity.com/paper/2202.12400/) (2022)
- [Layer Collapse Can be Induced by Unstructured Pruning](https://mlsanity.com/paper/2404.16890/) (2024)
- [Towards Efficient Capsule Networks](https://mlsanity.com/paper/2208.09203/) (2022)
- [The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth](https://mlsanity.com/paper/2404.18949/) (2024)
- [Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantization](https://mlsanity.com/paper/2306.11800/) (2023)
- [Find the Lady: Permutation and Re-Synchronization of Deep Neural Networks](https://mlsanity.com/paper/2312.14182/) (2023)
- [LaCoOT: Layer Collapse through Optimal Transport](https://mlsanity.com/paper/2406.08933/) (2024)
- [Routing the Lottery: Adaptive Subnetworks for Heterogeneous Data](https://mlsanity.com/paper/2601.22141/) (2026)
- [Playing the Lottery With Concave Regularizers for Sparse Trainable Neural Networks](https://mlsanity.com/paper/2501.11135/) (2025)
- [From Deep to Shallow: Unconstrained and Efficient Layer Merging Strategy](https://mlsanity.com/paper/2609.04881/) (2026)

## Abstract (arXiv)

Deep neural networks include millions of learnable parameters, making their deployment over resource-constrained devices problematic. SeReNe (Sensitivity-based Regularization of Neurons) is a method for learning sparse topologies with a structure, exploiting neural sensitivity as a regularizer. We define the sensitivity of a neuron as the variation of the network output with respect to the variation of the activity of the neuron. The lower the sensitivity of a neuron, the less the network output is perturbed if the neuron output changes. By including the neuron sensitivity in the cost function as a regularization term, we areable to prune neurons with low sensitivity. As entire neurons are pruned rather then single parameters, practical network footprint reduction becomes possible. Our experimental results on multiple network architectures and datasets yield competitive compression ratios with respect to state-of-the-art references.