# Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

arXiv: https://arxiv.org/abs/1705.07565 · HTML: https://mlsanity.com/paper/1705.07565/

## Key facts

- Cited by 221 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2017.

## Built on by (most cited first, 30 of 221)

- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [DepGraph: Towards Any Structural Pruning](https://mlsanity.com/paper/2301.12900/) (2023)
- [Structural Pruning for Diffusion Models](https://mlsanity.com/paper/2305.10924/) (2023)
- [Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better](https://mlsanity.com/paper/2106.08962/) (2021)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [Mobile Edge Intelligence for Large Language Models: A Contemporary Survey](https://mlsanity.com/paper/2407.18921/) (2024)
- [Monarch: Expressive Structured Matrices for Efficient and Accurate Training](https://mlsanity.com/paper/2204.00595/) (2022)
- [Unified Visual Transformer Compression](https://mlsanity.com/paper/2203.08243/) (2022)
- [Efficient Methods for Natural Language Processing: A Survey](https://mlsanity.com/paper/2209.00099/) (2022)
- [Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models](https://mlsanity.com/paper/2112.00029/) (2021)
- [Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?](https://mlsanity.com/paper/2106.02890/) (2021)
- [Simple Hardware-Efficient Long Convolutions for Sequence Modeling](https://mlsanity.com/paper/2302.06646/) (2023)
- [GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration](https://mlsanity.com/paper/2504.02692/) (2025)
- [AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks](https://mlsanity.com/paper/2106.12379/) (2021)
- [Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better](https://mlsanity.com/paper/2112.09824/) (2021)
- [SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation](https://mlsanity.com/paper/2202.07471/) (2022)
- [M-FAC: Efficient Matrix-Free Approximations of Second-Order Information](https://mlsanity.com/paper/2107.03356/) (2021)
- [LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2305.18403/) (2023)
- [Analytic Insights into Structure and Rank of Neural Network Hessian Maps](https://mlsanity.com/paper/2106.16225/) (2021)
- [Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning](https://mlsanity.com/paper/2404.00781/) (2024)
- [Dynamic Sparse Training for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04217/) (2021)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [Hessian-Aware Pruning and Optimal Neural Implant](https://mlsanity.com/paper/2101.08940/) (2021)
- [DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization](https://mlsanity.com/paper/2410.16942/) (2024)
- [Neural Mean Discrepancy for Efficient Out-of-Distribution Detection](https://mlsanity.com/paper/2104.11408/) (2021)
- [The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter](https://mlsanity.com/paper/2306.03805/) (2023)

## Abstract (arXiv)

How to develop slim and accurate deep neural networks has become crucial for real- world applications, especially for those employed in embedded systems. Though previous work along this research line has shown some promising results, most existing methods either fail to significantly compress a well-trained deep network or require a heavy retraining process for the pruned deep network to re-boost its prediction performance. In this paper, we propose a new layer-wise pruning method for deep neural networks. In our proposed method, parameters of each individual layer are pruned independently based on second order derivatives of a layer-wise error function with respect to the corresponding parameters. We prove that the final prediction performance drop after pruning is bounded by a linear combination of the reconstructed errors caused at each layer. Therefore, there is a guarantee that one only needs to perform a light retraining process on the pruned network to resume its original prediction performance. We conduct extensive experiments on benchmark datasets to demonstrate the effectiveness of our pruning method compared with several state-of-the-art baseline methods.