# Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?

arXiv: https://arxiv.org/abs/2308.06619 · HTML: https://mlsanity.com/paper/2308.06619/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 28 references that are in the map.
- First posted on arXiv 12 Aug 2023.

## Builds on (most cited first, 18 of 18)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Variational Dropout and the Local Reparameterization Trick](https://mlsanity.com/paper/1506.02557/) (2015)
- [The Lottery Ticket Hypothesis for Pre-trained BERT Networks](https://mlsanity.com/paper/2007.12223/) (2020)
- [Quantization Networks](https://mlsanity.com/paper/1911.09464/) (2019)
- [The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models](https://mlsanity.com/paper/2012.06908/) (2020)
- [Anomaly Detection in Autonomous Driving: A Survey](https://mlsanity.com/paper/2204.07974/) (2022)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [Packed-Ensembles for Efficient Uncertainty Estimation](https://mlsanity.com/paper/2210.09184/) (2022)
- [An Entropy-based Pruning Method for CNN Compression](https://mlsanity.com/paper/1706.05791/) (2017)
- [Sparse Double Descent: Where Network Pruning Aggravates Overfitting](https://mlsanity.com/paper/2206.08684/) (2022)
- [LOss-Based SensiTivity rEgulaRization: towards deep sparse neural networks](https://mlsanity.com/paper/2011.09905/) (2020)
- [2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy](https://mlsanity.com/paper/1809.02220/) (2018)
- [The rise of the lottery heroes: why zero-shot pruning is hard](https://mlsanity.com/paper/2202.12400/) (2022)
- [HEMP: High-order Entropy Minimization for neural network comPression](https://mlsanity.com/paper/2107.05298/) (2021)
- [Towards Efficient Capsule Networks](https://mlsanity.com/paper/2208.09203/) (2022)

## Built on by (most cited first, 20 of 20)

- [DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning](https://mlsanity.com/paper/2509.16105/) (2025)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models](https://mlsanity.com/paper/2507.18725/) (2025)
- [Layer Collapse Can be Induced by Unstructured Pruning](https://mlsanity.com/paper/2404.16890/) (2024)
- [The Simpler The Better: An Entropy-Based Importance Metric To Reduce Neural Networks' Depth](https://mlsanity.com/paper/2404.18949/) (2024)
- [3D Point Cloud Network Pruning: When Some Weights Do not Matter](https://mlsanity.com/paper/2408.14601/) (2024)
- [TOAST: Transformer Optimization using Adaptive and Simple Transformations](https://mlsanity.com/paper/2410.04941/) (2024)
- [When Does Pruning Benefit Vision Representations?](https://mlsanity.com/paper/2507.01722/) (2025)
- [LaCoOT: Layer Collapse through Optimal Transport](https://mlsanity.com/paper/2406.08933/) (2024)
- [NoWag: A Unified Framework for Shape Preserving Compression of Large Language Models](https://mlsanity.com/paper/2504.14569/) (2025)
- [Routing the Lottery: Adaptive Subnetworks for Heterogeneous Data](https://mlsanity.com/paper/2601.22141/) (2026)
- [NuWa: Deriving Lightweight Class-Specific Vision Transformers for Edge Devices](https://mlsanity.com/paper/2504.03118/) (2025)
- [RanDeS: Randomized Delta Superposition for Multi-Model Compression](https://mlsanity.com/paper/2505.11204/) (2025)
- [Towards Universal & Efficient Model Compression via Exponential Torque Pruning](https://mlsanity.com/paper/2506.22015/) (2025)
- [Beyond topography: Topographic regularization improves robustness and reshapes representations in convolutional neural networks](https://mlsanity.com/paper/2508.00043/) (2025)
- [Pruning and Quantization Impact on Graph Neural Networks](https://mlsanity.com/paper/2510.22058/) (2025)
- [Teacher-Guided One-Shot Pruning via Context-Aware Knowledge Distillation](https://mlsanity.com/paper/2511.16653/) (2025)
- [Joint Routing and Model Pruning for Decentralized Federated Learning in Bandwidth-Constrained Multi-Hop Wireless Networks](https://mlsanity.com/paper/2603.15188/) (2026)
- [CutClean: Neural Network Pruning for Privacy-Preserving Inference](https://mlsanity.com/paper/2608.13773/) (2026)
- [From Deep to Shallow: Unconstrained and Efficient Layer Merging Strategy](https://mlsanity.com/paper/2609.04881/) (2026)

## Abstract (arXiv)

Pruning is a widely used technique for reducing the size of deep neural networks while maintaining their performance. However, such a technique, despite being able to massively compress deep models, is hardly able to remove entire layers from a model (even when structured): is this an addressable task? In this study, we introduce EGP, an innovative Entropy Guided Pruning algorithm aimed at reducing the size of deep neural networks while preserving their performance. The key focus of EGP is to prioritize pruning connections in layers with low entropy, ultimately leading to their complete removal. Through extensive experiments conducted on popular models like ResNet-18 and Swin-T, our findings demonstrate that EGP effectively compresses deep neural networks while maintaining competitive performance levels. Our results not only shed light on the underlying mechanism behind the advantages of unstructured pruning, but also pave the way for further investigations into the intricate relationship between entropy, pruning techniques, and deep learning performance. The EGP algorithm and its insights hold great promise for advancing the field of network compression and optimization. The source code for EGP is released open-source.