# DropNet: Reducing Neural Network Complexity via Iterative Pruning

arXiv: https://arxiv.org/abs/2207.06646 · HTML: https://mlsanity.com/paper/2207.06646/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 38 references that are in the map.
- First posted on arXiv 14 Jul 2022.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Deep Learning with Limited Numerical Precision](https://mlsanity.com/paper/1502.02551/) (2015)
- [Structured Pruning of Deep Convolutional Neural Networks](https://mlsanity.com/paper/1512.08571/) (2015)
- [Learning the Number of Neurons in Deep Networks](https://mlsanity.com/paper/1611.06321/) (2016)

## Built on by (most cited first, 29 of 29)

- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Fluctuation-based Adaptive Structured Pruning for Large Language Models](https://mlsanity.com/paper/2312.11983/) (2023)
- [Empowering Edge Intelligence: A Comprehensive Survey on On-Device AI Models](https://mlsanity.com/paper/2503.06027/) (2025)
- [Reassessing Layer Pruning in LLMs: New Insights and Methods](https://mlsanity.com/paper/2411.15558/) (2024)
- [Pruning for Robust Concept Erasing in Diffusion Models](https://mlsanity.com/paper/2405.16534/) (2024)
- [A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services](https://mlsanity.com/paper/2304.14271/) (2023)
- [Relationship between Model Compression and Adversarial Robustness: A Review of Current Evidence](https://mlsanity.com/paper/2311.15782/) (2023)
- [FastFLUX: Pruning FLUX with Block-wise Replacement and Sandwich Training](https://mlsanity.com/paper/2506.10035/) (2025)
- [When Layers Play the Lottery, all Tickets Win at Initialization](https://mlsanity.com/paper/2301.10835/) (2023)
- [Approximate Computing Survey, Part II: Application-Specific & Architectural Approximation Techniques and Applications](https://mlsanity.com/paper/2307.11128/) (2023)
- [Benchmarking Adversarial Robustness of Compressed Deep Learning Models](https://mlsanity.com/paper/2308.08160/) (2023)
- [Neural Networks Reduction via Lumping](https://mlsanity.com/paper/2209.07475/) (2022)
- [COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models](https://mlsanity.com/paper/2210.15523/) (2022)
- [Mixture-of-Linguistic-Experts Adapters for Improving and Interpreting Pre-trained Language Models](https://mlsanity.com/paper/2310.16240/) (2023)
- [Lightweight Embedded FPGA Deployment of Learned Image Compression with Knowledge Distillation and Hybrid Quantization](https://mlsanity.com/paper/2503.04832/) (2025)
- [An Efficient Training Algorithm for Models with Block-wise Sparsity](https://mlsanity.com/paper/2503.21928/) (2025)
- [HASS: Hardware-Aware Sparsity Search for Dataflow DNN Accelerator](https://mlsanity.com/paper/2406.03088/) (2024)
- [Hyperpruning: Efficient Search through Pruned Variants of Recurrent Neural Networks Leveraging Lyapunov Spectrum](https://mlsanity.com/paper/2506.07975/) (2025)
- [Optimizing Learning Rate Schedules for Iterative Pruning of Deep Neural Networks](https://mlsanity.com/paper/2212.06144/) (2022)
- [Understanding Activation Patterns in Artificial Neural Networks by Exploring Stochastic Processes](https://mlsanity.com/paper/2308.00858/) (2023)
- [Equitable-FL: Federated Learning with Sparsity for Resource-Constrained Environment](https://mlsanity.com/paper/2309.00864/) (2023)
- [Dependable Distributed Training of Compressed Machine Learning Models](https://mlsanity.com/paper/2402.14346/) (2024)
- [Advancing IIoT with Over-the-Air Federated Learning: The Role of Iterative Magnitude Pruning](https://mlsanity.com/paper/2403.14120/) (2024)
- [A Comprehensive Study of Structural Pruning for Vision Models](https://mlsanity.com/paper/2406.12315/) (2024)
- [FedTSA: A Cluster-based Two-Stage Aggregation Method for Model-heterogeneous Federated Learning](https://mlsanity.com/paper/2407.05098/) (2024)
- [Exploring Vision Neural Network Pruning via Screening Methodology](https://mlsanity.com/paper/2502.07189/) (2025)
- [Vanishing Contributions: A Unified Framework for Smooth and Iterative Model Compression](https://mlsanity.com/paper/2510.09696/) (2025)
- [Regularisation in neural networks: a survey and empirical analysis of approaches](https://mlsanity.com/paper/2601.23131/) (2026)
- [S4oP: Operator-level Pruning of Structured State Space Models for Resource-Constrained Devices](https://mlsanity.com/paper/2606.18096/) (2026)

## Abstract (arXiv)

Modern deep neural networks require a significant amount of computing time and power to train and deploy, which limits their usage on edge devices. Inspired by the iterative weight pruning in the Lottery Ticket Hypothesis, we propose DropNet, an iterative pruning method which prunes nodes/filters to reduce network complexity. DropNet iteratively removes nodes/filters with the lowest average post-activation value across all training samples. Empirically, we show that DropNet is robust across diverse scenarios, including MLPs and CNNs using the MNIST, CIFAR-10 and Tiny ImageNet datasets. We show that up to 90% of the nodes/filters can be removed without any significant loss of accuracy. The final pruned network performs well even with reinitialization of the weights and biases. DropNet also has similar accuracy to an oracle which greedily removes nodes/filters one at a time to minimise training loss, highlighting its effectiveness.