# Towards Compact CNNs via Collaborative Compression

arXiv: https://arxiv.org/abs/2105.11228 · HTML: https://mlsanity.com/paper/2105.11228/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 24 May 2021.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression](https://mlsanity.com/paper/1707.06342/) (2017)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [NISP: Pruning Networks using Neuron Importance Score Propagation](https://mlsanity.com/paper/1711.05908/) (2017)
- [Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures](https://mlsanity.com/paper/1607.03250/) (2016)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Convolutional Networks with Adaptive Inference Graphs](https://mlsanity.com/paper/1711.11503/) (2017)
- [Towards Optimal Structured CNN Pruning via Generative Adversarial Learning](https://mlsanity.com/paper/1903.09291/) (2019)
- [CondenseNet: An Efficient DenseNet using Learned Group Convolutions](https://mlsanity.com/paper/1711.09224/) (2017)
- [Accelerating Very Deep Convolutional Networks for Classification and Detection](https://mlsanity.com/paper/1505.06798/) (2015)
- [Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure](https://mlsanity.com/paper/1904.03837/) (2019)

## Built on by (most cited first, 30 of 36)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [NOLA: Compressing LoRA using Linear Combination of Random Basis](https://mlsanity.com/paper/2310.02556/) (2023)
- [Parameter-Efficient Sparsity for Large Language Models Fine-Tuning](https://mlsanity.com/paper/2205.11005/) (2022)
- [Auto-Train-Once: Controller Network Guided Automatic Network Pruning from Scratch](https://mlsanity.com/paper/2403.14729/) (2024)
- [Trainability Preserving Neural Pruning](https://mlsanity.com/paper/2207.12534/) (2022)
- [Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruning](https://mlsanity.com/paper/2301.05219/) (2023)
- [Interpretations Steered Network Pruning via Amortized Inferred Saliency Maps](https://mlsanity.com/paper/2209.02869/) (2022)
- [ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks](https://mlsanity.com/paper/2401.10341/) (2024)
- [Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignment](https://mlsanity.com/paper/2403.19490/) (2024)
- [An Effective Information Theoretic Framework for Channel Pruning](https://mlsanity.com/paper/2408.16772/) (2024)
- [Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy](https://mlsanity.com/paper/2111.09635/) (2021)
- [Network Pruning via Feature Shift Minimization](https://mlsanity.com/paper/2207.02632/) (2022)
- [Influence Function Based Second-Order Channel Pruning-Evaluating True Loss Changes For Pruning Is Possible Without Retraining](https://mlsanity.com/paper/2308.06755/) (2023)
- [MCNC: Manifold-Constrained Reparameterization for Neural Compression](https://mlsanity.com/paper/2406.19301/) (2024)
- [Compact Model Training by Low-Rank Projection with Energy Transfer](https://mlsanity.com/paper/2204.05566/) (2022)
- [Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping](https://mlsanity.com/paper/2308.07641/) (2023)
- [Automatic Diagnosis of Myocarditis Disease in Cardiac MRI Modality using Deep Transformers and Explainable Artificial Intelligence](https://mlsanity.com/paper/2210.14611/) (2022)
- [Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler](https://mlsanity.com/paper/2307.00198/) (2023)
- [LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from Scratch](https://mlsanity.com/paper/2309.14157/) (2023)
- [Differentiable Model Scaling using Differentiable Topk](https://mlsanity.com/paper/2405.07194/) (2024)
- [Prioritized Subnet Sampling for Resource-Adaptive Supernet Training](https://mlsanity.com/paper/2109.05432/) (2021)
- [Evolving Transferable Neural Pruning Functions](https://mlsanity.com/paper/2110.10876/) (2021)
- [PRANC: Pseudo RAndom Networks for Compacting deep models](https://mlsanity.com/paper/2206.08464/) (2022)
- [A Differentiable Framework for End-to-End Learning of Hybrid Structured Compression](https://mlsanity.com/paper/2309.13077/) (2023)
- [Dynamic Shuffle: An Efficient Channel Mixture Method](https://mlsanity.com/paper/2310.02776/) (2023)
- [AutoSculpt: A Pattern-based Model Auto-pruning Framework Using Reinforcement Learning and Graph Learning](https://mlsanity.com/paper/2412.18091/) (2024)
- [One-Cycle Structured Pruning via Stability-Driven Subnetwork Search](https://mlsanity.com/paper/2501.13439/) (2025)
- [Enhanced Structured Lasso Pruning with Class-wise Information](https://mlsanity.com/paper/2502.09125/) (2025)
- [LegoNet: Memory Footprint Reduction Through Block Weight Clustering](https://mlsanity.com/paper/2603.06606/) (2026)

## Abstract (arXiv)

Channel pruning and tensor decomposition have received extensive attention in convolutional neural network compression. However, these two techniques are traditionally deployed in an isolated manner, leading to significant accuracy drop when pursuing high compression rates. In this paper, we propose a Collaborative Compression (CC) scheme, which joints channel pruning and tensor decomposition to compress CNN models by simultaneously learning the model sparsity and low-rankness. Specifically, we first investigate the compression sensitivity of each layer in the network, and then propose a Global Compression Rate Optimization that transforms the decision problem of compression rate into an optimization problem. After that, we propose multi-step heuristic compression to remove redundant compression units step-by-step, which fully considers the effect of the remaining compression space (i.e., unremoved compression units). Our method demonstrates superior performance gains over previous ones on various datasets and backbone architectures. For example, we achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.