mlsanityData to 7 Oct 2026

The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks

arXiv 2203.04466 ↗First posted 9 Mar 2022cs.LG cs.CVcited by 41

Not on the map. Why?

Abstract (arXiv)

Neural networks tend to achieve better accuracy with training if they are larger -- even if the resulting models are overparameterized. Nevertheless, carefully removing such excess parameters before, during, or after training may also produce models with similar or even improved accuracy. In many cases, that can be curiously achieved by heuristics as simple as removing a percentage of the weights with the smallest absolute value -- even though magnitude is not a perfect proxy for weight relevance. With the premise that obtaining significantly better performance from pruning depends on accounting for the combined effect of removing multiple weights, we revisit one of the classic approaches for impact-based pruning: the Optimal Brain Surgeon(OBS). We propose a tractable heuristic for solving the combinatorial extension of OBS, in which we select weights for simultaneous removal, as well as a systematic update of the remaining weights. Our selection method outperforms other methods under high sparsity, and the weight update is advantageous even when combined with the other methods.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 75 of its 112 references that are in the map. First posted on arXiv 9 Mar 2022.

Builds on

75 of 112 references.

References · 75

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  3. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  7. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  10. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
  11. 2018Rethinking the Value of Network Pruningcited by 595
  12. 2020What is the State of Neural Network Pruning?cited by 578
38 more
  1. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  2. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  3. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  4. 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
  5. 2018Stronger generalization bounds for deep nets via a compression approachcited by 306
  6. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
  7. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
  8. 2016Understanding Deep Neural Networks with Rectified Linear Unitscited by 253
  9. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  10. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  11. 2021Adversarial Neuron Pruning Purifies Backdoored Deep Modelscited by 230
  12. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  13. 2013Predicting Parameters in Deep Learningcited by 223
  14. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  15. 2021Dataset Distillation with Infinitely Wide Convolutional Networkscited by 215
  16. 2019Deconstructing Lottery Tickets: Zeros, Signs, and the Supermaskcited by 206
  17. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 200
  18. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
  19. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  20. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 181
  21. 2019Complexity of Linear Regions in Deep Networkscited by 166
  22. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  23. 2020Characterising Bias in Compressed Modelscited by 162
  24. 2017Bounding and Counting Linear Regions of Deep Neural Networkscited by 161
  25. 2019Deep ReLU Networks Have Surprisingly Few Activation Patternscited by 160
  26. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  27. 2015Data-free parameter pruning for Deep Neural Networkscited by 112
  28. 2019EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasiscited by 98
  29. 2019Provable Filter Pruning for Efficient Neural Networkscited by 98
  30. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  31. 2013On the number of response regions of deep feed forward networks with piece-wise linear activationscited by 94
  32. 2015Representation Benefits of Deep Feedforward Networkscited by 84
  33. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  34. 2015Fast ConvNets Using Group-wise Brain Damagecited by 81
  35. 2020Optimal Lottery Tickets via SubsetSum: Logarithmic Over-Parameterization is Sufficientcited by 81
  36. 2020Logarithmic Pruning is All You Needcited by 71
  37. 2020Good Subnetworks Provably Exist: Pruning via Greedy Forward Selectioncited by 70
  38. 2021Only Train Once: A One-Shot Neural Network Training And Pruning Frameworkcited by 70

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 11; 2024: 14; 2025: 6; 2026: 5.

41 papers.

Sort:

By area:

  1. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  2. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 45
  3. 2023Fast as CHITA: Neural Network Pruning with Combinatorial Optimizationcited by 40
  4. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
  5. 2022Why Random Pruning Is All We Need to Start Sparsecited by 26
  6. 2024LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss-error-aware Gridcited by 20
  7. 2024ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Modelscited by 18
  8. 2024OSSCAR: One-Shot Structured Pruning in Vision and Language Models with Combinatorial Optimizationcited by 17
  9. 2024MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruningcited by 13
  10. 2024FALCON: FLOP-Aware Combinatorial Optimization for Neural Network Pruningcited by 11
  11. 2024Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Frameworkcited by 10
  12. 2022Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithmcited by 9
29 more
  1. 2022SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevancecited by 9
  2. 2023How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspectivecited by 9
  3. 2024Jointly Training and Pruning CNNs via Learnable Agent Guidance and Alignmentcited by 9
  4. 2023Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regionscited by 8
  5. 2024The Iterative Optimal Brain Surgeon: Faster Sparse Recovery by Leveraging Second-Order Informationcited by 8
  6. 2024SNP: Structured Neuron-level Pruning to Preserve Attention Scorescited by 5
  7. 2024MINI-LLM: Memory-Efficient Structured Pruning for Large Language Modelscited by 5
  8. 2025Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representationscited by 5
  9. 2025SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shotcited by 5
  10. 2023UPSCALE: Unconstrained Channel Pruningcited by 4
  11. 2023Influence Function Based Second-Order Channel Pruning-Evaluating True Loss Changes For Pruning Is Possible Without Retrainingcited by 4
  12. 2024A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizationscited by 4
  13. 2024Free Lunch in the Forest: Functionally-Identical Pruning of Boosted Tree Ensemblescited by 4
  14. 2023Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solvercited by 3
  15. 2025Optimal Brain Apoptosiscited by 3
  16. 2023Model-based feature selection for neural networks: A mixed-integer programming approachcited by 2
  17. 2023SWAP: Sparse Entropic Wasserstein Regression for Robust Network Pruningcited by 2
  18. 2025Data-Free Pruning of Self-Attention Layers in LLMscited by 2
  19. 2022Multi-Objective Linear Ensembles for Robust and Sparse Training of Few-Bit Neural Networkscited by 1
  20. 2023Importance Estimation with Random Gradient for Neural Network Pruningcited by 1
  21. 2025Theoretical Compression Bounds for Wide Multilayer Perceptronscited by 1
  22. 2026LLM Compression by Block Removal with Constrained Binary Optimizationcited by 1
  23. 2023Biologically-Motivated Learning Model for Instructed Visual Processingcited by 0
  24. 2024A constrained optimization approach to improve robustness of neural networkscited by 0
  25. 2025Optimization over Trained (and Sparse) Neural Networks: A Surrogate within a Surrogatecited by 0
  26. 2026Ethical Fairness in Ubiquitous Health Sensing without Known Attributescited by 0
  27. 2026STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healingcited by 0
  28. 2026Saturation Makes Quantization Error Additive: A Coverage Model with a Certificatecited by 0
  29. 2026Resource-Efficient Pruning for Transformer via Low-Rank Importance Estimationcited by 0