# Manifold Regularized Dynamic Network Pruning

arXiv: https://arxiv.org/abs/2103.05861 · HTML: https://mlsanity.com/paper/2103.05861/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 64 references that are in the map.
- First posted on arXiv 10 Mar 2021.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [Image reconstruction by domain transform manifold learning](https://mlsanity.com/paper/1704.08841/) (2017)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [Quantized Convolutional Neural Networks for Mobile Devices](https://mlsanity.com/paper/1512.06473/) (2015)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Discrimination-aware Channel Pruning for Deep Neural Networks](https://mlsanity.com/paper/1810.11809/) (2018)
- [Towards Optimal Structured CNN Pruning via Generative Adversarial Learning](https://mlsanity.com/paper/1903.09291/) (2019)
- [Dynamic Channel Pruning: Feature Boosting and Suppression](https://mlsanity.com/paper/1810.05331/) (2018)
- [Provable Filter Pruning for Efficient Neural Networks](https://mlsanity.com/paper/1911.07412/) (2019)
- [Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification](https://mlsanity.com/paper/1611.05377/) (2016)
- [SCOP: Scientific Control for Reliable Neural Network Pruning](https://mlsanity.com/paper/2010.10732/) (2020)
- [Channel Gating Neural Networks](https://mlsanity.com/paper/1805.12549/) (2018)
- [CARS: Continuous Evolution for Efficient Neural Architecture Search](https://mlsanity.com/paper/1909.04977/) (2019)
- [Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression](https://mlsanity.com/paper/2003.08935/) (2020)
- [GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet](https://mlsanity.com/paper/2003.11236/) (2020)
- [More is Less: A More Complicated Network with Less Inference Complexity](https://mlsanity.com/paper/1703.08651/) (2017)

## Built on by (most cited first, 30 of 43)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Patch Slimming for Efficient Vision Transformers](https://mlsanity.com/paper/2106.02852/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Dynamic Resolution Network](https://mlsanity.com/paper/2106.02898/) (2021)
- [Understanding the Dynamics of DNNs Using Graph Modularity](https://mlsanity.com/paper/2111.12485/) (2021)
- [Carrying out CNN Channel Pruning in a White Box](https://mlsanity.com/paper/2104.11883/) (2021)
- [RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Merging](https://mlsanity.com/paper/2110.01397/) (2021)
- [Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs](https://mlsanity.com/paper/2203.07808/) (2022)
- [Efficient Stein Variational Inference for Reliable Distribution-lossless Network Pruning](https://mlsanity.com/paper/2212.03537/) (2022)
- [Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models](https://mlsanity.com/paper/2406.12042/) (2024)
- [SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance](https://mlsanity.com/paper/2207.04089/) (2022)
- [System-status-aware Adaptive Network for Online Streaming Video Understanding](https://mlsanity.com/paper/2303.15742/) (2023)
- [GradMDM: Adversarial Attack on Dynamic Networks](https://mlsanity.com/paper/2304.06724/) (2023)
- [Hadamax Encoding: Elevating Performance in Model-Free Atari](https://mlsanity.com/paper/2505.15345/) (2025)
- [An Empirical Study of Adder Neural Networks for Object Detection](https://mlsanity.com/paper/2112.13608/) (2021)
- [Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning](https://mlsanity.com/paper/2203.02651/) (2022)
- [Dynamic Structure Pruning for Compressing CNNs](https://mlsanity.com/paper/2303.09736/) (2023)
- [On Accelerating Edge AI: Optimizing Resource-Constrained Environments](https://mlsanity.com/paper/2501.15014/) (2025)
- [Progressive Channel-Shrinking Network](https://mlsanity.com/paper/2304.00280/) (2023)
- [RedTest: Towards Measuring Redundancy in Deep Neural Networks Effectively](https://mlsanity.com/paper/2411.10507/) (2024)
- [T'yr-the-Pruner: Structural Pruning LLMs via Global Sparsity Distribution Optimization](https://mlsanity.com/paper/2503.09657/) (2025)
- [BiMLP: Compact Binary Architectures for Vision Multi-Layer Perceptrons](https://mlsanity.com/paper/2212.14158/) (2022)
- [Boosting Convolutional Neural Networks with Middle Spectrum Grouped Convolution](https://mlsanity.com/paper/2304.06305/) (2023)
- [Semantic Alignment and Reinforcement for Data-Free Quantization of Vision Transformers](https://mlsanity.com/paper/2412.16553/) (2024)
- [Pick-or-Mix: Dynamic Channel Sampling for ConvNets](https://mlsanity.com/paper/2406.10935/) (2024)
- [Deep Lookup Network](https://mlsanity.com/paper/2509.13662/) (2025)
- [Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks](https://mlsanity.com/paper/2108.13728/) (2021)
- [A Differentiable Framework for End-to-End Learning of Hybrid Structured Compression](https://mlsanity.com/paper/2309.13077/) (2023)

## Abstract (arXiv)

Neural network pruning is an essential approach for reducing the computational complexity of deep models so that they can be well deployed on resource-limited devices. Compared with conventional methods, the recently developed dynamic pruning methods determine redundant filters variant to each input instance which achieves higher acceleration. Most of the existing methods discover effective sub-networks for each instance independently and do not utilize the relationship between different inputs. To maximally excavate redundancy in the given network architecture, this paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks (dubbed as ManiDP). We first investigate the recognition complexity and feature similarity between images in the training set. Then, the manifold relationship between instances and the pruned sub-networks will be aligned in the training procedure. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost compared to the state-of-the-art methods. For example, our method can reduce 55.3% FLOPs of ResNet-34 with only 0.57% top-1 accuracy degradation on ImageNet.