# WeightMom: Learning Sparse Networks using Iterative Momentum-based pruning

arXiv: https://arxiv.org/abs/2208.05970 · HTML: https://mlsanity.com/paper/2208.05970/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Builds on (most cited first, 14 of 14)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [A Survey of Neuromorphic Computing and Neural Networks in Hardware](https://mlsanity.com/paper/1705.06963/) (2017)
- [Exploring Sparsity in Recurrent Neural Networks](https://mlsanity.com/paper/1704.05119/) (2017)
- [Global Sparse Momentum SGD for Pruning Very Deep Neural Networks](https://mlsanity.com/paper/1909.12778/) (2019)
- [Hardware-Aware Machine Learning: Modeling and Optimization](https://mlsanity.com/paper/1809.05476/) (2018)
- [N2NSkip: Learning Highly Sparse Networks using Neuron-to-Neuron Skip Connections](https://mlsanity.com/paper/2208.03662/) (2022)
- [Spectral Reflectance based Heart Rate Measurement from Facial Video](https://mlsanity.com/paper/1901.00273/) (2019)
- [A neuromorphic approach to image processing and machine vision](https://mlsanity.com/paper/2209.02595/) (2022)
- [Implementation of fast ICA using memristor crossbar arrays for blind image source separations](https://mlsanity.com/paper/2208.04317/) (2022)

## Built on by (most cited first, 1 of 1)

- [Model Explainability in Physiological and Healthcare-based Neural Networks](https://mlsanity.com/paper/2304.14495/) (2023)

## Abstract (arXiv)

Deep Neural Networks have been used in a wide variety of applications with significant success. However, their highly complex nature owing to comprising millions of parameters has lead to problems during deployment in pipelines with low latency requirements. As a result, it is more desirable to obtain lightweight neural networks which have the same performance during inference time. In this work, we propose a weight based pruning approach in which the weights are pruned gradually based on their momentum of the previous iterations. Each layer of the neural network is assigned an importance value based on their relative sparsity, followed by the magnitude of the weight in the previous iterations. We evaluate our approach on networks such as AlexNet, VGG16 and ResNet50 with image classification datasets such as CIFAR-10 and CIFAR-100. We found that the results outperformed the previous approaches with respect to accuracy and compression ratio. Our method is able to obtain a compression of 15% for the same degradation in accuracy on both the datasets.