# DHP: Differentiable Meta Pruning via HyperNetworks

arXiv: https://arxiv.org/abs/2003.13683 · HTML: https://mlsanity.com/paper/2003.13683/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Mar 2020.

## Built on by (most cited first, 30 of 57)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [The Ninth NTIRE 2024 Efficient Super-Resolution Challenge Report](https://mlsanity.com/paper/2404.10343/) (2024)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [Lightweight Adaptive Feature De-drifting for Compressed Image Classification](https://mlsanity.com/paper/2401.01724/) (2024)
- [Blueprint Separable Residual Network for Efficient Image Super-Resolution](https://mlsanity.com/paper/2205.05996/) (2022)
- [Manifold Regularized Dynamic Network Pruning](https://mlsanity.com/paper/2103.05861/) (2021)
- [NTIRE 2022 Challenge on Efficient Super-Resolution: Methods and Results](https://mlsanity.com/paper/2205.05675/) (2022)
- [Learning the Effect of Registration Hyperparameters with HyperMorph](https://mlsanity.com/paper/2203.16680/) (2022)
- [Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Search](https://mlsanity.com/paper/2108.08910/) (2021)
- [Topology-Aware Network Pruning using Multi-stage Graph Embedding and Reinforcement Learning](https://mlsanity.com/paper/2102.03214/) (2021)
- [Fast and Memory-Efficient Network Towards Efficient Image Super-Resolution](https://mlsanity.com/paper/2204.08397/) (2022)
- [CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolution](https://mlsanity.com/paper/2207.10345/) (2022)
- [Compiler-Aware Neural Architecture Search for On-Mobile Real-time Super-Resolution](https://mlsanity.com/paper/2207.12577/) (2022)
- [Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch](https://mlsanity.com/paper/2403.14729/) (2024)
- [Towards Efficient Graph Convolutional Networks for Point Cloud Handling](https://mlsanity.com/paper/2104.05706/) (2021)
- [Edge-enhanced Feature Distillation Network for Efficient Super-Resolution](https://mlsanity.com/paper/2204.08759/) (2022)
- [Lightweight Pixel Difference Networks for Efficient Visual Representation Learning](https://mlsanity.com/paper/2402.00422/) (2024)
- [Towards Lightweight Super-Resolution with Dual Regression Learning](https://mlsanity.com/paper/2207.07929/) (2022)
- [Toward DNN of LUTs: Learning Efficient Image Restoration with Multiple Look-Up Tables](https://mlsanity.com/paper/2303.14506/) (2023)
- [AdaBM: On-the-Fly Adaptive Bit Mapping for Image Super-Resolution](https://mlsanity.com/paper/2404.03296/) (2024)
- [InsertNeRF: Instilling Generalizability into NeRF with HyperNet Modules](https://mlsanity.com/paper/2308.13897/) (2023)
- [EZCrop: Energy-Zoned Channels for Robust Output Pruning](https://mlsanity.com/paper/2105.03679/) (2021)
- [CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization](https://mlsanity.com/paper/2110.10921/) (2021)
- [Fast Conditional Network Compression Using Bayesian HyperNetworks](https://mlsanity.com/paper/2205.06404/) (2022)
- [Dynamic Structure Pruning for Compressing CNNs](https://mlsanity.com/paper/2303.09736/) (2023)
- [Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks](https://mlsanity.com/paper/2307.13337/) (2023)
- [On Accelerating Edge AI: Optimizing Resource-Constrained Environments](https://mlsanity.com/paper/2501.15014/) (2025)
- [Overparametrization of HyperNetworks at Fixed FLOP-Count Enables Fast Neural Image Enhancement](https://mlsanity.com/paper/2105.08470/) (2021)

## Abstract (arXiv)

Network pruning has been the driving force for the acceleration of neural networks and the alleviation of model storage/transmission burden. With the advent of AutoML and neural architecture search (NAS), pruning has become topical with automatic mechanism and searching based architecture optimization. Yet, current automatic designs rely on either reinforcement learning or evolutionary algorithm. Due to the non-differentiability of those algorithms, the pruning algorithm needs a long searching stage before reaching the convergence. To circumvent this problem, this paper introduces a differentiable pruning method via hypernetworks for automatic network pruning. The specifically designed hypernetworks take latent vectors as input and generate the weight parameters of the backbone network. The latent vectors control the output channels of the convolutional layers in the backbone network and act as a handle for the pruning of the layers. By enforcing ℓ1 sparsity regularization to the latent vectors and utilizing proximal gradient solver, sparse latent vectors can be obtained. Passing the sparsified latent vectors through the hypernetworks, the corresponding slices of the generated weight parameters can be removed, achieving the effect of network pruning. The latent vectors of all the layers are pruned together, resulting in an automatic layer configuration. Extensive experiments are conducted on various networks for image classification, single image super-resolution, and denoising. And the experimental results validate the proposed method.