# ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting

arXiv: https://arxiv.org/abs/2007.03260 · HTML: https://mlsanity.com/paper/2007.03260/

## Key facts

- Cited by 85 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jul 2020.

## Built on by (most cited first, 30 of 85)

- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs](https://mlsanity.com/paper/2203.06717/) (2022)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Towards Efficient Visual Adaption via Structural Re-parameterization](https://mlsanity.com/paper/2302.08106/) (2023)
- [UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition](https://mlsanity.com/paper/2311.15599/) (2023)
- [Revisiting Random Channel Pruning for Neural Network Compression](https://mlsanity.com/paper/2205.05676/) (2022)
- [EVC: Towards Real-Time Neural Image Compression with Mask Decay](https://mlsanity.com/paper/2302.05071/) (2023)
- [Mixture-of-Subspaces in Low-Rank Adaptation](https://mlsanity.com/paper/2406.11909/) (2024)
- [Improved Implicit Neural Representation with Fourier Reparameterized Training](https://mlsanity.com/paper/2401.07402/) (2024)
- [Re-parameterizing Your Optimizers rather than Architectures](https://mlsanity.com/paper/2205.15242/) (2022)
- [RepMLPNet: Hierarchical Vision MLP with Re-parameterized Locality](https://mlsanity.com/paper/2112.11081/) (2021)
- [An Incremental Unified Framework for Small Defect Inspection](https://mlsanity.com/paper/2312.08917/) (2023)
- [Carrying out CNN Channel Pruning in a White Box](https://mlsanity.com/paper/2104.11883/) (2021)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [Estimator Meets Equilibrium Perspective: A Rectified Straight Through Estimator for Binary Neural Networks Training](https://mlsanity.com/paper/2308.06689/) (2023)
- [Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptation](https://mlsanity.com/paper/2305.00603/) (2023)
- [Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruning](https://mlsanity.com/paper/2301.05219/) (2023)
- [E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity](https://mlsanity.com/paper/2310.15929/) (2023)
- [DiSparse: Disentangled Sparsification for Multitask Model Compression](https://mlsanity.com/paper/2206.04662/) (2022)
- [Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report](https://mlsanity.com/paper/2211.03885/) (2022)
- [RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixer](https://mlsanity.com/paper/2304.05659/) (2023)
- [Experts Weights Averaging: A New General Training Scheme for Vision Transformers](https://mlsanity.com/paper/2308.06093/) (2023)
- [Lottery Jackpots Exist in Pre-trained Models](https://mlsanity.com/paper/2104.08700/) (2021)
- [Practical Network Acceleration with Tiny Sets](https://mlsanity.com/paper/2202.07861/) (2022)
- [CP3: Channel Pruning Plug-in for Point-based Networks](https://mlsanity.com/paper/2303.13097/) (2023)
- [Performance-aware Approximation of Global Channel Pruning for Multitask CNNs](https://mlsanity.com/paper/2303.11923/) (2023)
- [CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs](https://mlsanity.com/paper/2309.15755/) (2023)
- [RefConv: Re-parameterized Refocusing Convolution for Powerful ConvNets](https://mlsanity.com/paper/2310.10563/) (2023)
- [Weight-Inherited Distillation for Task-Agnostic BERT Compression](https://mlsanity.com/paper/2305.09098/) (2023)
- [Structurally Prune Anything: Any Architecture, Any Framework, Any Time](https://mlsanity.com/paper/2403.18955/) (2024)

## Abstract (arXiv)

We propose ResRep, a novel method for lossless channel pruning (a.k.a. filter pruning), which slims down a CNN by reducing the width (number of output channels) of convolutional layers. Inspired by the neurobiology research about the independence of remembering and forgetting, we propose to re-parameterize a CNN into the remembering parts and forgetting parts, where the former learn to maintain the performance and the latter learn to prune. Via training with regular SGD on the former but a novel update rule with penalty gradients on the latter, we realize structured sparsity. Then we equivalently merge the remembering and forgetting parts into the original architecture with narrower layers. In this sense, ResRep can be viewed as a successful application of Structural Re-parameterization. Such a methodology distinguishes ResRep from the traditional learning-based pruning paradigm that applies a penalty on parameters to produce sparsity, which may suppress the parameters essential for the remembering. ResRep slims down a standard ResNet-50 with 76.15% accuracy on ImageNet to a narrower one with only 45% FLOPs and no accuracy drop, which is the first to achieve lossless pruning with such a high compression ratio. The code and models are at https://github.com/DingXiaoH/ResRep.