# To Filter Prune, or to Layer Prune, That Is The Question

arXiv: https://arxiv.org/abs/2007.05667 · HTML: https://mlsanity.com/paper/2007.05667/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jul 2020.

## Built on by (most cited first, 23 of 23)

- [LAPTOP-Diff: Layer Pruning and Normalized Distillation for Compressing Diffusion Models](https://mlsanity.com/paper/2404.11098/) (2024)
- [DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization](https://mlsanity.com/paper/2410.16942/) (2024)
- [Understanding the Dynamics of DNNs Using Graph Modularity](https://mlsanity.com/paper/2111.12485/) (2021)
- [SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models](https://mlsanity.com/paper/2410.14720/) (2024)
- [Fire Together Wire Together: A Dynamic Pruning Approach with Self-Supervised Mask Prediction](https://mlsanity.com/paper/2110.08232/) (2021)
- [DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks](https://mlsanity.com/paper/2206.00843/) (2022)
- [A Generic Layer Pruning Method for Signal Modulation Recognition Deep Learning Models](https://mlsanity.com/paper/2406.07929/) (2024)
- [AGIC: Approximate Gradient Inversion Attack on Federated Learning](https://mlsanity.com/paper/2204.13784/) (2022)
- [SR-init: An interpretable layer pruning method](https://mlsanity.com/paper/2303.07677/) (2023)
- [UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer](https://mlsanity.com/paper/2401.06426/) (2024)
- [LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging](https://mlsanity.com/paper/2406.12837/) (2024)
- [Restructurable Activation Networks](https://mlsanity.com/paper/2208.08562/) (2022)
- [Temporal Action Detection Model Compression by Progressive Block Drop](https://mlsanity.com/paper/2503.16916/) (2025)
- [MDP: Multidimensional Vision Model Pruning with Latency Constraint](https://mlsanity.com/paper/2504.02168/) (2025)
- [Multi-Dimensional Pruning: Joint Channel, Layer and Block Pruning with Latency Constraint](https://mlsanity.com/paper/2406.12079/) (2024)
- [FCOS: A Two-Stage Recoverable Model Pruning Framework for Automatic Modulation Recognition](https://mlsanity.com/paper/2505.21571/) (2025)
- [Block Pruning for Enhanced Efficiency in Convolutional Neural Networks](https://mlsanity.com/paper/2312.16904/) (2023)
- [Rethinking Theoretical Illumination for Efficient Low-Light Image Enhancement](https://mlsanity.com/paper/2409.05274/) (2024)
- [Straightforward Layer-wise Pruning for More Efficient Visual Adaptation](https://mlsanity.com/paper/2407.14330/) (2024)
- [An Adaptive Tensor-Train Decomposition Approach for Efficient Deep Neural Network Compression](https://mlsanity.com/paper/2408.01534/) (2024)
- [Towards Universal & Efficient Model Compression via Exponential Torque Pruning](https://mlsanity.com/paper/2506.22015/) (2025)
- [Learn&Drop: Fast Learning of CNNs based on Layer Dropping](https://mlsanity.com/paper/2604.23403/) (2026)
- [Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference](https://mlsanity.com/paper/2609.05275/) (2026)

## Abstract (arXiv)

Recent advances in pruning of neural networks have made it possible to remove a large number of filters or weights without any perceptible drop in accuracy. The number of parameters and that of FLOPs are usually the reported metrics to measure the quality of the pruned models. However, the gain in speed for these pruned models is often overlooked in the literature due to the complex nature of latency measurements. In this paper, we show the limitation of filter pruning methods in terms of latency reduction and propose LayerPrune framework. LayerPrune presents a set of layer pruning methods based on different criteria that achieve higher latency reduction than filter pruning methods on similar accuracy. The advantage of layer pruning over filter pruning in terms of latency reduction is a result of the fact that the former is not constrained by the original model's depth and thus allows for a larger range of latency reduction. For each filter pruning method we examined, we use the same filter importance criterion to calculate a per-layer importance score in one-shot. We then prune the least important layers and fine-tune the shallower model which obtains comparable or better accuracy than its filter-based pruning counterpart. This one-shot process allows to remove layers from single path networks like VGG before fine-tuning, unlike in iterative filter pruning, a minimum number of filters per layer is required to allow for data flow which constraint the search space. To the best of our knowledge, we are the first to examine the effect of pruning methods on latency metric instead of FLOPs for multiple networks, datasets and hardware targets. LayerPrune also outperforms handcrafted architectures such as Shufflenet, MobileNet, MNASNet and ResNet18 by 7.3%, 4.6%, 2.8% and 0.5% respectively on similar latency budget on ImageNet dataset.