Data to 7 Oct 2026

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

arXiv 1705.07565 ↗First posted 22 May 2017cs.NE cs.CV cs.LGcited by 219

Not on the map. Why?

Abstract (arXiv)

How to develop slim and accurate deep neural networks has become crucial for real- world applications, especially for those employed in embedded systems. Though previous work along this research line has shown some promising results, most existing methods either fail to significantly compress a well-trained deep network or require a heavy retraining process for the pruned deep network to re-boost its prediction performance. In this paper, we propose a new layer-wise pruning method for deep neural networks. In our proposed method, parameters of each individual layer are pruned independently based on second order derivatives of a layer-wise error function with respect to the corresponding parameters. We prove that the final prediction performance drop after pruning is bounded by a linear combination of the reconstructed errors caused at each layer. Therefore, there is a guarantee that one only needs to perform a light retraining process on the pruned network to resume its original prediction performance. We conduct extensive experiments on benchmark datasets to demonstrate the effectiveness of our pruning method compared with several state-of-the-art baseline methods.

Key facts. Cited by 219 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 48; 2022: 41; 2023: 46; 2024: 33; 2025: 37; 2026: 14.

219 papers.

Sort:

By area:

2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
2023DepGraph: Towards Any Structural Pruningcited by 202
2023Structural Pruning for Diffusion Modelscited by 193
2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
2021Learned Token Pruning for Transformerscited by 138
2021Group Fisher Pruning for Practical Network Compressioncited by 130
2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
2022Monarch: Expressive Structured Matrices for Efficient and Accurate Trainingcited by 100
2022Unified Visual Transformer Compressioncited by 79

38 more

2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
2021AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networkscited by 58
2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
2021Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Bettercited by 54
2025GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibrationcited by 54
2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
2021Analytic Insights into Structure and Rank of Neural Network Hessian Mapscited by 52
2022SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximationcited by 51
2021Dynamic Sparse Training for Deep Reinforcement Learningcited by 49
2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 49
2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
2021Hessian-Aware Pruning and Optimal Neural Implantcited by 43
2024DiP-GO: A Diffusion Pruner via Few-step Gradient Optimizationcited by 43
2021Neural Mean Discrepancy for Efficient Out-of-Distribution Detectioncited by 42
2023The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Mattercited by 42
2022The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networkscited by 41
2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
2023SEENN: Towards Temporal Spiking Early-Exit Neural Networkscited by 39
2021MQBench: Towards Reproducible and Deployable Model Quantization Benchmarkcited by 38
2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
2024Lightweight Deep Learning for Resource-Constrained Environments: A Surveycited by 38
2023LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recoverycited by 37
2021Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Searchcited by 34
2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
2023FlashFFTConv: Efficient Convolutions for Long Sequences with Tensor Corescited by 32
2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
2022RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratchcited by 30
2023NTK-SAP: Improving neural network pruning by aligning training dynamicscited by 30
2023QuantEase: Optimization-based Quantization for Language Modelscited by 28
2023Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!cited by 27
2024Differentiable Weightless Neural Networkscited by 27
2022Why Random Pruning Is All We Need to Start Sparsecited by 26
2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
2024SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Modelscited by 24
2023Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspectivecited by 23

50 most cited of 219 All 219 (JSON)