# GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization

arXiv: https://arxiv.org/abs/2109.02220 · HTML: https://mlsanity.com/paper/2109.02220/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 88 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 30 of 71)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)

## Built on by (most cited first, 29 of 29)

- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Unified Visual Transformer Compression](https://mlsanity.com/paper/2203.08243/) (2022)
- [Advancing Model Pruning via Bi-level Optimization](https://mlsanity.com/paper/2210.04092/) (2022)
- [Structural Pruning via Latency-Saliency Knapsack](https://mlsanity.com/paper/2210.06659/) (2022)
- [CHEX: CHannel EXploration for CNN Model Compression](https://mlsanity.com/paper/2203.15794/) (2022)
- [Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization](https://mlsanity.com/paper/2206.01198/) (2022)
- [Quarantine: Sparsity Can Uncover the Trojan Attack Trigger for Free](https://mlsanity.com/paper/2205.11819/) (2022)
- [Automatic Attention Pruning: Improving and Automating Model Pruning using Attentions](https://mlsanity.com/paper/2303.08595/) (2023)
- [SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance](https://mlsanity.com/paper/2207.04089/) (2022)
- [FoldGPT: Simple and Effective Large Language Model Compression Scheme](https://mlsanity.com/paper/2407.00928/) (2024)
- [Ensemble Knowledge Guided Sub-network Search and Fine-tuning for Filter Pruning](https://mlsanity.com/paper/2203.02651/) (2022)
- [REPrune: Channel Pruning via Kernel Representative Selection](https://mlsanity.com/paper/2402.17862/) (2024)
- [decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points](https://mlsanity.com/paper/2404.12759/) (2024)
- [Adaptive Activation-based Structured Pruning](https://mlsanity.com/paper/2201.10520/) (2022)
- [How to Train Unstable Looped Tensor Network](https://mlsanity.com/paper/2203.02617/) (2022)
- [Resource Constrained Model Compression via Minimax Optimization for Spiking Neural Networks](https://mlsanity.com/paper/2308.04672/) (2023)
- [RdimKD: Generic Distillation Paradigm by Dimensionality Reduction](https://mlsanity.com/paper/2312.08700/) (2023)
- [Filter Pruning based on Information Capacity and Independence](https://mlsanity.com/paper/2303.03645/) (2023)
- [AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networks](https://mlsanity.com/paper/2304.06941/) (2023)
- [Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler](https://mlsanity.com/paper/2307.00198/) (2023)
- [Differentiable Model Scaling using Differentiable Topk](https://mlsanity.com/paper/2405.07194/) (2024)
- [USDC: Unified Static and Dynamic Compression for Visual Transformer](https://mlsanity.com/paper/2310.11117/) (2023)
- [Enhanced Structured Lasso Pruning with Class-wise Information](https://mlsanity.com/paper/2502.09125/) (2025)
- [MaskPrune: Mask-based LLM Pruning for Layer-wise Uniform Structures](https://mlsanity.com/paper/2502.14008/) (2025)
- [FPG-NAS: FLOPs-Aware Gated Differentiable Neural Architecture Search for Efficient 6DoF Pose Estimation](https://mlsanity.com/paper/2508.03618/) (2025)
- [CWP: Instance complexity weighted channel-wise soft masks for network pruning](https://mlsanity.com/paper/2209.03534/) (2022)
- [A Generalization of Continuous Relaxation in Structured Pruning](https://mlsanity.com/paper/2308.14605/) (2023)
- [Archtree: on-the-fly tree-structured exploration for latency-aware pruning of deep neural networks](https://mlsanity.com/paper/2311.10549/) (2023)
- [Catalyst: a Novel Regularizer for Structured Pruning with Auxiliary Extension of Parameter Space](https://mlsanity.com/paper/2507.14170/) (2025)

## Abstract (arXiv)

Model compression techniques are recently gaining explosive attention for obtaining efficient AI models for various real-time applications. Channel pruning is one important compression strategy and is widely used in slimming various DNNs. Previous gate-based or importance-based pruning methods aim to remove channels whose importance is smallest. However, it remains unclear what criteria the channel importance should be measured on, leading to various channel selection heuristics. Some other sampling-based pruning methods deploy sampling strategies to train sub-nets, which often causes the training instability and the compressed model's degraded performance. In view of the research gaps, we present a new module named Gates with Differentiable Polarization (GDP), inspired by principled optimization ideas. GDP can be plugged before convolutional layers without bells and whistles, to control the on-and-off of each channel or whole layer block. During the training process, the polarization effect will drive a subset of gates to smoothly decrease to exact zero, while other gates gradually stay away from zero by a large margin. When training terminates, those zero-gated channels can be painlessly removed, while other non-zero gates can be absorbed into the succeeding convolution kernel, causing completely no interruption to training nor damage to the trained model. Experiments conducted over CIFAR-10 and ImageNet datasets show that the proposed GDP algorithm achieves the state-of-the-art performance on various benchmark DNNs at a broad range of pruning ratios. We also apply GDP to DeepLabV3Plus-ResNet50 on the challenging Pascal VOC segmentation task, whose test performance sees no drop (even slightly improved) with over 60% FLOPs saving.