# Boosting Pruned Networks with Linear Over-parameterization

arXiv: https://arxiv.org/abs/2204.11444 · HTML: https://mlsanity.com/paper/2204.11444/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 35 references that are in the map.
- First posted on arXiv 25 Apr 2022.

## Builds on (most cited first, 29 of 29)

- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [NISP: Pruning Networks using Neuron Importance Score Propagation](https://mlsanity.com/paper/1711.05908/) (2017)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks](https://mlsanity.com/paper/1908.03930/) (2019)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers](https://mlsanity.com/paper/1804.03294/) (2018)
- [Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1909.08174/) (2019)
- [DENSE: Data-Free One-Shot Federated Learning](https://mlsanity.com/paper/2112.12371/) (2021)
- [A Signal Propagation Perspective for Pruning Neural Networks at Initialization](https://mlsanity.com/paper/1906.06307/) (2019)
- [ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting](https://mlsanity.com/paper/2007.03260/) (2020)
- [EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning](https://mlsanity.com/paper/2007.02491/) (2020)
- [Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure](https://mlsanity.com/paper/1904.03837/) (2019)
- [ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks](https://mlsanity.com/paper/1811.10495/) (2018)
- [SpArch: Efficient Architecture for Sparse Matrix Multiplication](https://mlsanity.com/paper/2002.08947/) (2020)
- [DiracNets: Training Very Deep Neural Networks Without Skip-Connections](https://mlsanity.com/paper/1706.00388/) (2017)
- [DO-Conv: Depthwise Over-parameterized Convolutional Layer](https://mlsanity.com/paper/2006.12030/) (2020)
- [Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework](https://mlsanity.com/paper/2010.04879/) (2020)
- [Learning to Design Circuits](https://mlsanity.com/paper/1812.02734/) (2018)

## Built on by (most cited first, 4 of 4)

- [SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models](https://mlsanity.com/paper/2410.14720/) (2024)
- [Enhanced Pruning Strategy for Multi-Component Neural Architectures Using Component-Aware Graph Analysis](https://mlsanity.com/paper/2504.13296/) (2025)
- [Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression](https://mlsanity.com/paper/2604.04988/) (2026)
- [COMponent-Aware Pruning for Accelerated Control Tasks in Latent Space Models](https://mlsanity.com/paper/2508.08144/) (2025)

## Abstract (arXiv)

Structured pruning compresses neural networks by reducing channels (filters) for fast inference and low footprint at run-time. To restore accuracy after pruning, fine-tuning is usually applied to pruned networks. However, too few remaining parameters in pruned networks inevitably bring a great challenge to fine-tuning to restore accuracy. To address this challenge, we propose a novel method that first linearly over-parameterizes the compact layers in pruned networks to enlarge the number of fine-tuning parameters and then re-parameterizes them to the original layers after fine-tuning. Specifically, we equivalently expand the convolution/linear layer with several consecutive convolution/linear layers that do not alter the current output feature maps. Furthermore, we utilize similarity-preserving knowledge distillation that encourages the over-parameterized block to learn the immediate data-to-data similarities of the corresponding dense layer to maintain its feature learning ability. The proposed method is comprehensively evaluated on CIFAR-10 and ImageNet which significantly outperforms the vanilla fine-tuning strategy, especially for large pruning ratio.