# Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures

arXiv: https://arxiv.org/abs/1607.03250 · HTML: https://mlsanity.com/paper/1607.03250/

## Key facts

- Cited by 221 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2016.

## Built on by (most cited first, 30 of 221)

- [A Simple and Effective Pruning Approach for Large Language Models](https://mlsanity.com/paper/2306.11695/) (2023)
- [Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild](https://mlsanity.com/paper/2401.13627/) (2024)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement](https://mlsanity.com/paper/2304.01195/) (2023)
- [Continual Backprop: Stochastic Gradient Descent with Persistent Randomness](https://mlsanity.com/paper/2108.06325/) (2021)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [Convolutional Neural Network Pruning with Structural Redundancy Reduction](https://mlsanity.com/paper/2104.03438/) (2021)
- [Only Train Once: A One-Shot Neural Network Training And Pruning Framework](https://mlsanity.com/paper/2107.07467/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Pufferfish: Communication-efficient Models At No Extra Cost](https://mlsanity.com/paper/2103.03936/) (2021)
- [Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better](https://mlsanity.com/paper/2112.09824/) (2021)
- [Dynamic Sparse Training for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04217/) (2021)
- [Online Continual Learning for Embedded Devices](https://mlsanity.com/paper/2203.10681/) (2022)
- [Towards Accurate Post-training Quantization for Diffusion Models](https://mlsanity.com/paper/2305.18723/) (2023)
- [Q-VLM: Post-training Quantization for Large Vision-Language Models](https://mlsanity.com/paper/2410.08119/) (2024)
- [Maintaining Plasticity in Deep Continual Learning](https://mlsanity.com/paper/2306.13812/) (2023)
- [Structural Pruning via Latency-Saliency Knapsack](https://mlsanity.com/paper/2210.06659/) (2022)
- [NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks](https://mlsanity.com/paper/2206.09117/) (2022)
- [Towards Compact CNNs via Collaborative Compression](https://mlsanity.com/paper/2105.11228/) (2021)
- [Pruning-as-Search: Efficient Neural Architecture Search via Channel Pruning and Structural Reparameterization](https://mlsanity.com/paper/2206.01198/) (2022)
- [OTOV2: Automatic, Generic, User-Friendly](https://mlsanity.com/paper/2303.06862/) (2023)
- [RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch](https://mlsanity.com/paper/2205.15043/) (2022)
- [DECORE: Deep Compression with Reinforcement Learning](https://mlsanity.com/paper/2106.06091/) (2021)
- [Content-Aware GAN Compression](https://mlsanity.com/paper/2104.02244/) (2021)
- [Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets](https://mlsanity.com/paper/2202.04736/) (2022)
- [A Machine Learning-oriented Survey on Tiny Machine Learning](https://mlsanity.com/paper/2309.11932/) (2023)

## Abstract (arXiv)

State-of-the-art neural networks are getting deeper and wider. While their performance increases with the increasing number of layers and neurons, it is crucial to design an efficient deep architecture in order to reduce computational and memory costs. Designing an efficient neural network, however, is labor intensive requiring many experiments, and fine-tunings. In this paper, we introduce network trimming which iteratively optimizes the network by pruning unimportant neurons based on analysis of their outputs on a large dataset. Our algorithm is inspired by an observation that the outputs of a significant portion of neurons in a large network are mostly zero, regardless of what inputs the network received. These zero activation neurons are redundant, and can be removed without affecting the overall accuracy of the network. After pruning the zero activation neurons, we retrain the network using the weights before pruning as initialization. We alternate the pruning and retraining to further reduce zero activations in a network. Our experiments on the LeNet and VGG-16 show that we can achieve high compression ratio of parameters without losing or even achieving higher accuracy than the original network.