# Dynamic Network Surgery for Efficient DNNs

arXiv: https://arxiv.org/abs/1608.04493 · HTML: https://mlsanity.com/paper/1608.04493/

## Key facts

- Cited by 233 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2016.

## Built on by (most cited first, 30 of 233)

- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Continual Pre-training of Language Models](https://mlsanity.com/paper/2302.03241/) (2023)
- [FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout](https://mlsanity.com/paper/2102.13451/) (2021)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training](https://mlsanity.com/paper/2102.02887/) (2021)
- [Distilling Object Detectors via Decoupled Features](https://mlsanity.com/paper/2103.14475/) (2021)
- [MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge](https://mlsanity.com/paper/2110.14032/) (2021)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [MagMax: Leveraging Model Merging for Seamless Continual Learning](https://mlsanity.com/paper/2407.06322/) (2024)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021)
- [Dynamic Sparse Training for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04217/) (2021)
- [ALISA: Accelerating Large Language Model Inference via Sparsity-Aware KV Caching](https://mlsanity.com/paper/2403.17312/) (2024)
- [Differentiable Subset Pruning of Transformer Heads](https://mlsanity.com/paper/2108.04657/) (2021)
- [The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation](https://mlsanity.com/paper/2110.03036/) (2021)
- [Effective Sparsification of Neural Networks with Global Sparsity Constraint](https://mlsanity.com/paper/2105.01571/) (2021)
- [ZeroFL: Efficient On-Device Training for Federated Learning with Local Sparsity](https://mlsanity.com/paper/2208.02507/) (2022)
- [Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations](https://mlsanity.com/paper/2205.13571/) (2022)
- [Recent Advances on Neural Network Pruning at Initialization](https://mlsanity.com/paper/2103.06460/) (2021)
- [Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI](https://mlsanity.com/paper/2111.06061/) (2021)
- [Lightweight Deep Learning for Resource-Constrained Environments: A Survey](https://mlsanity.com/paper/2404.07236/) (2024)
- [CHEX: CHannel EXploration for CNN Model Compression](https://mlsanity.com/paper/2203.15794/) (2022)
- [Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Search](https://mlsanity.com/paper/2108.08910/) (2021)
- [Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications](https://mlsanity.com/paper/2204.11786/) (2022)
- [Topology-Aware Network Pruning using Multi-stage Graph Embedding and Reinforcement Learning](https://mlsanity.com/paper/2102.03214/) (2021)
- [Efficient Neural Network Training via Forward and Backward Propagation Sparsification](https://mlsanity.com/paper/2111.05685/) (2021)

## Abstract (arXiv)

Deep learning has become a ubiquitous technology to improve machine intelligence. However, most of the existing deep models are structurally very complex, making them difficult to be deployed on the mobile platforms with limited computational power. In this paper, we propose a novel network compression method called dynamic network surgery, which can remarkably reduce the network complexity by making on-the-fly connection pruning. Unlike the previous methods which accomplish this task in a greedy way, we properly incorporate connection splicing into the whole process to avoid incorrect pruning and make it as a continual network maintenance. The effectiveness of our method is proved with experiments. Without any accuracy loss, our method can efficiently compress the number of parameters in LeNet-5 and AlexNet by a factor of bm108× and bm17.7× respectively, proving that it outperforms the recent pruning method by considerable margins. Code and some models are available at https://github.com/yiwenguo/Dynamic-Network-Surgery.