# SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning

arXiv: https://arxiv.org/abs/2110.11395 · HTML: https://mlsanity.com/paper/2110.11395/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 42 references that are in the map.
- First posted on arXiv 19 Oct 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [Pruning neural networks without any data by iteratively conserving synaptic flow](https://mlsanity.com/paper/2006.05467/) (2020)
- [Picking Winning Tickets Before Training by Preserving Gradient Flow](https://mlsanity.com/paper/2002.07376/) (2020)
- [AMC: AutoML for Model Compression and Acceleration on Mobile Devices](https://mlsanity.com/paper/1802.03494/) (2018)
- [Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration](https://mlsanity.com/paper/1811.00250/) (2018)
- [HRank: Filter Pruning using High-Rank Feature Map](https://mlsanity.com/paper/2002.10179/) (2020)
- [Towards Optimal Structured CNN Pruning via Generative Adversarial Learning](https://mlsanity.com/paper/1903.09291/) (2019)
- [Group Fisher Pruning for Practical Network Compression](https://mlsanity.com/paper/2108.00708/) (2021)
- [EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis](https://mlsanity.com/paper/1905.05934/) (2019)
- [Faster gaze prediction with dense networks and Fisher pruning](https://mlsanity.com/paper/1801.05787/) (2018)
- [SCOP: Scientific Control for Reliable Neural Network Pruning](https://mlsanity.com/paper/2010.10732/) (2020)
- [EagleEye: Fast Sub-net Evaluation for Efficient Neural Network Pruning](https://mlsanity.com/paper/2007.02491/) (2020)
- [Towards Efficient Model Compression via Learned Global Ranking](https://mlsanity.com/paper/1904.12368/) (2019)
- [DMCP: Differentiable Markov Channel Pruning for Neural Networks](https://mlsanity.com/paper/2005.03354/) (2020)
- [Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot](https://mlsanity.com/paper/2009.11094/) (2020)
- [AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates](https://mlsanity.com/paper/1907.03141/) (2019)
- [Robust Pruning at Initialization](https://mlsanity.com/paper/2002.08797/) (2020)
- [ASAP: Architecture Search, Anneal and Prune](https://mlsanity.com/paper/1904.04123/) (2019)
- [Locally Free Weight Sharing for Network Width Search](https://mlsanity.com/paper/2102.05258/) (2021)

## Built on by (most cited first, 30 of 30)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [A Survey of Lottery Ticket Hypothesis](https://mlsanity.com/paper/2403.04861/) (2024)
- [A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts](https://mlsanity.com/paper/2405.16646/) (2024)
- [Structured Pruning Adapters](https://mlsanity.com/paper/2211.10155/) (2022)
- [SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance](https://mlsanity.com/paper/2207.04089/) (2022)
- [On the Opportunities of Green Computing: A Survey](https://mlsanity.com/paper/2311.00447/) (2023)
- [The Right to be Forgotten in Pruning: Unveil Machine Unlearning on Sparse Models](https://mlsanity.com/paper/2507.18725/) (2025)
- [Effective Layer Pruning Through Similarity Metric Perspective](https://mlsanity.com/paper/2405.17081/) (2024)
- [Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities](https://mlsanity.com/paper/2602.00693/) (2026)
- [Mutual Information Preserving Neural Network Pruning](https://mlsanity.com/paper/2411.00147/) (2024)
- [Influence Function Based Second-Order Channel Pruning-Evaluating True Loss Changes For Pruning Is Possible Without Retraining](https://mlsanity.com/paper/2308.06755/) (2023)
- [REPrune: Channel Pruning via Kernel Representative Selection](https://mlsanity.com/paper/2402.17862/) (2024)
- [Elastic ViTs from Pretrained Models without Retraining](https://mlsanity.com/paper/2510.17700/) (2025)
- [Optimal Brain Connection: Towards Efficient Structural Pruning](https://mlsanity.com/paper/2508.05521/) (2025)
- [LAPP: Layer Adaptive Progressive Pruning for Compressing CNNs from Scratch](https://mlsanity.com/paper/2309.14157/) (2023)
- [Test-time Cost-and-Quality Controllable Arbitrary-Scale Super-Resolution with Variable Fourier Components](https://mlsanity.com/paper/2412.05517/) (2024)
- [Maximum Redundancy Pruning: A Principle-Driven Layerwise Sparsity Allocation for LLMs](https://mlsanity.com/paper/2503.18377/) (2025)
- [AutoSculpt: A Pattern-based Model Auto-pruning Framework Using Reinforcement Learning and Graph Learning](https://mlsanity.com/paper/2412.18091/) (2024)
- [Hardware-Aware DNN Compression for Homogeneous Edge Devices](https://mlsanity.com/paper/2501.15240/) (2025)
- [DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs](https://mlsanity.com/paper/2502.12455/) (2025)
- [Train Large, Deploy Compact: Structured Compression for Compact Low-Rank Adaptation](https://mlsanity.com/paper/2510.00192/) (2025)
- [(PASS) Visual Prompt Locates Good Structure Sparsity through a Recurrent HyperNetwork](https://mlsanity.com/paper/2407.17412/) (2024)
- [A Greedy Hierarchical Approach to Whole-Network Filter-Pruning in CNNs](https://mlsanity.com/paper/2409.03777/) (2024)
- [Layer Pruning with Consensus: A Triple-Win Solution](https://mlsanity.com/paper/2411.14345/) (2024)
- [Smooth Model Compression without Fine-Tuning](https://mlsanity.com/paper/2505.24469/) (2025)
- [Pruning Everything, Everywhere, All at Once](https://mlsanity.com/paper/2506.04513/) (2025)
- [Explore and Establish Synergistic Effects Between Weight Pruning and Coreset Selection in Neural Network Training](https://mlsanity.com/paper/2511.09901/) (2025)
- [Automatic Pruning Discovery for Large Language Models](https://mlsanity.com/paper/2511.15390/) (2025)
- [PruneShift: A Framework for Evaluating Decision Reliability in Structured Pruning](https://mlsanity.com/paper/2608.29765/) (2026)

## Abstract (arXiv)

Pruning neural networks reduces inference time and memory costs. On standard hardware, these benefits will be especially prominent if coarse-grained structures, like feature maps, are pruned. We devise two novel saliency-based methods for second-order structured pruning (SOSP) which include correlations among all structures and layers. Our main method SOSP-H employs an innovative second-order approximation, which enables saliency evaluations by fast Hessian-vector products. SOSP-H thereby scales like a first-order method despite taking into account the full Hessian. We validate SOSP-H by comparing it to our second method SOSP-I that uses a well-established Hessian approximation, and to numerous state-of-the-art methods. While SOSP-H performs on par or better in terms of accuracy, it has clear advantages in terms of scalability and efficiency. This allowed us to scale SOSP-H to large-scale vision tasks, even though it captures correlations across all layers of the network. To underscore the global nature of our pruning methods, we evaluate their performance not only by removing structures from a pretrained network, but also by detecting architectural bottlenecks. We show that our algorithms allow to systematically reveal architectural bottlenecks, which we then remove to further increase the accuracy of the networks.