# Learning Instance-wise Sparsity for Accelerating Deep Models

arXiv: https://arxiv.org/abs/1907.11840 · HTML: https://mlsanity.com/paper/1907.11840/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jul 2019.

## Built on by (most cited first, 7 of 7)

- [Dynamic Neural Networks: A Survey](https://mlsanity.com/paper/2102.04906/) (2021)
- [Manifold Regularized Dynamic Network Pruning](https://mlsanity.com/paper/2103.05861/) (2021)
- [GhostNets on Heterogeneous Devices via Cheap Operations](https://mlsanity.com/paper/2201.03297/) (2022)
- [Dynamic Slimmable Denoising Network](https://mlsanity.com/paper/2110.08940/) (2021)
- [Multi-stage feature decorrelation constraints for improving CNN classification performance](https://mlsanity.com/paper/2308.12880/) (2023)
- [ExpertSim: Fast Particle Detector Simulation Using Mixture-of-Generative-Experts](https://mlsanity.com/paper/2508.20991/) (2025)
- [Adaptive Structured Pruning of Convolutional Neural Networks for Time Series Classification](https://mlsanity.com/paper/2602.12744/) (2026)

## Abstract (arXiv)

Exploring deep convolutional neural networks of high efficiency and low memory usage is very essential for a wide variety of machine learning tasks. Most of existing approaches used to accelerate deep models by manipulating parameters or filters without data, e.g., pruning and decomposition. In contrast, we study this problem from a different perspective by respecting the difference between data. An instance-wise feature pruning is developed by identifying informative features for different instances. Specifically, by investigating a feature decay regularization, we expect intermediate feature maps of each instance in deep neural networks to be sparse while preserving the overall network performance. During online inference, subtle features of input images extracted by intermediate layers of a well-trained neural network can be eliminated to accelerate the subsequent calculations. We further take coefficient of variation as a measure to select the layers that are appropriate for acceleration. Extensive experiments conducted on benchmark datasets and networks demonstrate the effectiveness of the proposed method.