# Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework

arXiv: https://arxiv.org/abs/2411.18376 · HTML: https://mlsanity.com/paper/2411.18376/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 27 Nov 2024.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Train faster, generalize better: Stability of stochastic gradient descent](https://mlsanity.com/paper/1509.01240/) (2015)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning](https://mlsanity.com/paper/2208.11580/) (2022)
- [Comparing Rewinding and Fine-tuning in Neural Network Pruning](https://mlsanity.com/paper/2003.02389/) (2020)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks](https://mlsanity.com/paper/1805.12076/) (2018)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis](https://mlsanity.com/paper/1905.05934/) (2019)
- [A Stochastic Quasi-Newton Method for Large-Scale Optimization](https://mlsanity.com/paper/1401.7020/) (2014)
- [The Difficulty of Training Sparse Neural Networks](https://mlsanity.com/paper/1906.10732/) (2019)
- [M-FAC: Efficient Matrix-Free Approximations of Second-Order Information](https://mlsanity.com/paper/2107.03356/) (2021)
- [The Combinatorial Brain Surgeon: Pruning Weights That Cancel One Another in Neural Networks](https://mlsanity.com/paper/2203.04466/) (2022)
- [A Unified Pruning Framework for Vision Transformers](https://mlsanity.com/paper/2111.15127/) (2021)
- [A Survey of Methods for Low-Power Deep Learning and Computer Vision](https://mlsanity.com/paper/2003.11066/) (2020)
- [Learning Best Combination for Efficient N:M Sparsity](https://mlsanity.com/paper/2206.06662/) (2022)
- [CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Models](https://mlsanity.com/paper/2210.09223/) (2022)
- [ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models](https://mlsanity.com/paper/2406.07831/) (2024)
- [Dynamic N:M Fine-grained Structured Sparse Attention Mechanism](https://mlsanity.com/paper/2203.00091/) (2022)
- [Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning](https://mlsanity.com/paper/2209.14624/) (2022)
- [Training Recipe for N:M Structured Sparsity with Decaying Pruning Mask](https://mlsanity.com/paper/2209.07617/) (2022)

## Built on by (most cited first, 11 of 11)

- [HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs](https://mlsanity.com/paper/2502.00899/) (2025)
- [HEART-VIT: Hessian-Guided Efficient Dynamic Attention and Token Pruning in Vision Transformer](https://mlsanity.com/paper/2512.20120/) (2025)
- [Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems](https://mlsanity.com/paper/2502.14305/) (2025)
- [CORP: Closed-Form One-shot Representation-Preserving Structured Pruning for Transformers](https://mlsanity.com/paper/2602.05243/) (2026)
- [Cut-ViT: Task-Specific Model Pruning via Gram Anchoring Subspace Consistency](https://mlsanity.com/paper/2608.28205/) (2026)
- [TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Masks](https://mlsanity.com/paper/2505.23949/) (2025)
- [Winning the Lottery by Preserving Network Training Dynamics with Concrete Ticket Search](https://mlsanity.com/paper/2512.07142/) (2025)
- [STARFISH: faST Accuracy Recovery in pruned networks From Internal State Healing](https://mlsanity.com/paper/2606.01126/) (2026)
- [Pruning Deep Neural Networks via the Marchenko--Pastur Distribution](https://mlsanity.com/paper/2606.02608/) (2026)
- [Artificial Structure Function Search: Preserving Artificial Functional Connectivity for Structured Pruning](https://mlsanity.com/paper/2609.24401/) (2026)
- [Retraining-Free Depth Pruning of Vision Transformers: A Cross Domain Study of Layer Redundancy](https://mlsanity.com/paper/2610.15005/) (2026)

## Abstract (arXiv)

We present SNOWS, a one-shot post-training pruning framework aimed at reducing the cost of vision network inference without retraining. Current leading one-shot pruning methods minimize layer-wise least squares reconstruction error which does not take into account deeper network representations. We propose to optimize a more global reconstruction objective. This objective accounts for nonlinear activations deep in the network to obtain a better proxy for the network loss. This nonlinear objective leads to a more challenging optimization problem -- we demonstrate it can be solved efficiently using a specialized second-order optimization framework. A key innovation of our framework is the use of Hessian-free optimization to compute exact Newton descent steps without needing to compute or store the full Hessian matrix. A distinct advantage of SNOWS is that it can be readily applied on top of any sparse mask derived from prior methods, readjusting their weights to exploit nonlinearities in deep feature representations. SNOWS obtains state-of-the-art results on various one-shot pruning benchmarks including residual networks and Vision Transformers (ViT/B-16 and ViT/L-16, 86m and 304m parameters respectively).