# UPDP: A Unified Progressive Depth Pruner for CNN and Vision Transformer

arXiv: https://arxiv.org/abs/2401.06426 · HTML: https://mlsanity.com/paper/2401.06426/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 52 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Builds on (most cited first, 30 of 44)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Are Sixteen Heads Really Better than One?](https://mlsanity.com/paper/1905.10650/) (2019)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning](https://mlsanity.com/paper/1903.10258/) (2019)
- [Universally Slimmable Networks and Improved Training Techniques](https://mlsanity.com/paper/1903.05134/) (2019)
- [BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models](https://mlsanity.com/paper/2003.11142/) (2020)
- [Patch Slimming for Efficient Vision Transformers](https://mlsanity.com/paper/2106.02852/) (2021)

## Built on by (most cited first, 9 of 9)

- [The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2404.10343/) (2024)
- [PRANCE: Joint Token-Optimization and Structural Channel-Pruning for Adaptive ViT Inference](https://mlsanity.com/paper/2407.05010/) (2024)
- [Temporal Action Detection Model Compression by Progressive Block Drop](https://mlsanity.com/paper/2503.16916/) (2025)
- [MDP: Multidimensional Vision Model Pruning with Latency Constraint](https://mlsanity.com/paper/2504.02168/) (2025)
- [Adaptive Pruning of Pretrained Transformer via Differential Inclusions](https://mlsanity.com/paper/2501.03289/) (2025)
- [Reduced storage direct tensor ring decomposition for convolutional neural networks compression](https://mlsanity.com/paper/2405.10802/) (2024)
- [Layer Pruning with Consensus: A Triple-Win Solution](https://mlsanity.com/paper/2411.14345/) (2024)
- [Compact Vision Transformer by Reduction of Kernel Complexity](https://mlsanity.com/paper/2507.12780/) (2025)
- [HiAP: A Multi-Granular Stochastic Auto-Pruning Framework for Vision Transformers](https://mlsanity.com/paper/2603.12222/) (2026)

## Abstract (arXiv)

Traditional channel-wise pruning methods by reducing network channels struggle to effectively prune efficient CNN models with depth-wise convolutional layers and certain efficient modules, such as popular inverted residual blocks. Prior depth pruning methods by reducing network depths are not suitable for pruning some efficient models due to the existence of some normalization layers. Moreover, finetuning subnet by directly removing activation layers would corrupt the original model weights, hindering the pruned model from achieving high performance. To address these issues, we propose a novel depth pruning method for efficient models. Our approach proposes a novel block pruning strategy and progressive training method for the subnet. Additionally, we extend our pruning method to vision transformer models. Experimental results demonstrate that our method consistently outperforms existing depth pruning methods across various pruning configurations. We obtained three pruned ConvNeXtV1 models with our method applying on ConvNeXtV1, which surpass most SOTA efficient models with comparable inference performance. Our method also achieves state-of-the-art pruning performance on the vision transformer model.