# Revisiting Loss Modelling for Unstructured Pruning

arXiv: https://arxiv.org/abs/2006.12279 · HTML: https://mlsanity.com/paper/2006.12279/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jun 2020.

## Built on by (most cited first, 6 of 6)

- [Unified Data-Free Compression: Pruning and Quantization without Fine-Tuning](https://mlsanity.com/paper/2308.07209/) (2023)
- [On Efficient Variants of Segment Anything Model: A Survey](https://mlsanity.com/paper/2410.04960/) (2024)
- [Cyclical Pruning for Sparse Neural Networks](https://mlsanity.com/paper/2202.01290/) (2022)
- [Sparse then Prune: Toward Efficient Vision Transformers](https://mlsanity.com/paper/2307.11988/) (2023)
- [Data-Driven Low-Rank Neural Network Compression](https://mlsanity.com/paper/2107.05787/) (2021)
- [Person Detection Using an Ultra Low-resolution Thermal Imager on a Low-cost MCU](https://mlsanity.com/paper/2212.08415/) (2022)

## Abstract (arXiv)

By removing parameters from deep neural networks, unstructured pruning methods aim at cutting down memory footprint and computational cost, while maintaining prediction accuracy. In order to tackle this otherwise intractable problem, many of these methods model the loss landscape using first or second order Taylor expansions to identify which parameters can be discarded. We revisit loss modelling for unstructured pruning: we show the importance of ensuring locality of the pruning steps. We systematically compare first and second order Taylor expansions and empirically show that both can reach similar levels of performance. Finally, we show that better preserving the original network function does not necessarily transfer to better performing networks after fine-tuning, suggesting that only considering the impact of pruning on the loss might not be a sufficient objective to design good pruning criteria.