# WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

arXiv: https://arxiv.org/abs/2004.14340 · HTML: https://mlsanity.com/paper/2004.14340/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Apr 2020.

## Built on by (most cited first, 21 of 21)

- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy](https://mlsanity.com/paper/2103.03014/) (2021)
- [SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization](https://mlsanity.com/paper/2106.02266/) (2021)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [SSSE: Efficiently Erasing Samples from Trained Machine Learning Models](https://mlsanity.com/paper/2107.03860/) (2021)
- [Dual Lottery Ticket Hypothesis](https://mlsanity.com/paper/2203.04248/) (2022)
- [Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions](https://mlsanity.com/paper/2209.08554/) (2022)
- [ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models](https://mlsanity.com/paper/2310.02998/) (2023)
- [Trainability Preserving Neural Pruning](https://mlsanity.com/paper/2207.12534/) (2022)
- [Why is the State of Neural Network Pruning so Confusing? On the Fairness, Comparison Setup, and Trainability in Network Pruning](https://mlsanity.com/paper/2301.05219/) (2023)
- [Neural Network Compression for Noisy Storage Devices](https://mlsanity.com/paper/2102.07725/) (2021)
- [Is Oracle Pruning the True Oracle?](https://mlsanity.com/paper/2412.00143/) (2024)
- [Cyclical Pruning for Sparse Neural Networks](https://mlsanity.com/paper/2202.01290/) (2022)
- [SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shot](https://mlsanity.com/paper/2506.09613/) (2025)
- [Deep Neural Compression Via Concurrent Pruning and Self-Distillation](https://mlsanity.com/paper/2109.15014/) (2021)
- [HEART-VIT: Hessian-Guided Efficient Dynamic Attention and Token Pruning in Vision Transformer](https://mlsanity.com/paper/2512.20120/) (2025)
- [Dynamical Isometry: The Missing Ingredient for Neural Network Pruning](https://mlsanity.com/paper/2105.05916/) (2021)
- [A Granger-Causal Perspective on Gradient Descent with Application to Pruning](https://mlsanity.com/paper/2412.03035/) (2024)
- [UWC: Unit-wise Calibration Towards Rapid Network Compression](https://mlsanity.com/paper/2201.06376/) (2022)
- [Takeuchi's Information Criteria as Generalization Measures for DNNs Close to NTK Regime](https://mlsanity.com/paper/2602.23219/) (2026)
- [SparseForge: Efficient Semi-Structured LLM Sparsification via Annealing of Hessian-Guided Soft-Mask](https://mlsanity.com/paper/2605.06402/) (2026)

## Abstract (arXiv)

Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to neural network compression, where we build on the classic Optimal Brain Damage/Surgeon framework. We demonstrate that WoodFisher significantly outperforms popular state-of-the-art methods for one-shot pruning. Further, even when iterative, gradual pruning is considered, our method results in a gain in test accuracy over the state-of-the-art approaches, for pruning popular neural networks (like ResNet-50, MobileNetV1) trained on standard image classification datasets such as ImageNet ILSVRC. We examine how our method can be extended to take into account first-order information, as well as illustrate its ability to automatically set layer-wise pruning thresholds and perform compression in the limited-data regime. The code is available at the following link, https://github.com/IST-DASLab/WoodFisher.