mlsanityData to 7 Oct 2026

Neural network relief: a pruning algorithm based on neural activity

arXiv 2109.10795 ↗First posted 22 Sep 2021cs.LG cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

Current deep neural networks (DNNs) are overparameterized and use most of their neuronal connections during inference for each task. The human brain, however, developed specialized regions for different tasks and performs inference with a small fraction of its neuronal connections. We propose an iterative pruning strategy introducing a simple importance-score metric that deactivates unimportant connections, tackling overparameterization in DNNs and modulating the firing patterns. The aim is to find the smallest number of connections that is still capable of solving a given task with comparable accuracy, i.e. a simpler subnetwork. We achieve comparable performance for LeNet architectures on MNIST, and significantly higher parameter compression than state-of-the-art algorithms for VGG and ResNet architectures on CIFAR-10/100 and Tiny-ImageNet. Our approach also performs well for the two different optimizers considered -- Adam and SGD. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations, although it performs reasonably when compared to the state of the art.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 22 Sep 2021.

Builds on

32 of 50 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  6. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  10. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  11. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  12. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 697
20 more
  1. 2019Importance Estimation for Neural Network Pruningcited by 531
  2. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438
  3. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 313
  4. 2020HRank: Filter Pruning using High-Rank Feature Mapcited by 291
  5. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  6. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  7. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  8. 2017NISP: Pruning Networks using Neuron Importance Score Propagationcited by 228
  9. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 220
  10. 2017Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeoncited by 219
  11. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  12. 2017Bayesian Compression for Deep Learningcited by 116
  13. 2018Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layerscited by 112
  14. 2017Soft Weight-Sharing for Neural Network Compressioncited by 99
  15. 2019Adversarial Robustness vs Model Compression, or Both?cited by 98
  16. 2015Fast ConvNets Using Group-wise Brain Damagecited by 81
  17. 2019How Can We Be So Dense? The Benefits of Using Highly Sparse Representationscited by 45
  18. 2020Achieving Adversarial Robustness via Sparsitycited by 14
  19. 2018On Implicit Filter Level Sparsity in Convolutional Neural Networkscited by 9
  20. 2020Shapley Value as Principled Metric for Structured Network Pruningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 2; 2023: 3; 2024: 2; 2025: 1; 2026: 1.

10 papers.

  1. 2024Everybody Prune Now: Structured Pruning of LLMs with only Forward Passescited by 42
  2. 2022Continual Prune-and-Select: Class-incremental learning with specialized subnetworkscited by 16
  3. 2021Avoiding Forgetting and Allowing Forward Transfer in Continual Learning via Sparse Networkscited by 7
  4. 2022Cooperative data-driven modelingcited by 5
  5. 2023iPINNs: Incremental learning for Physics-informed neural networkscited by 3
  6. 2023Learning a Consensus Sub-Network with Polarization Regularization and One Pass Trainingcited by 1
  7. 2023Continual learning for surface defect segmentation by subnetwork creation and selectioncited by 1
  8. 2025Linearity-based neural network compressioncited by 1
  9. 2024Reduced storage direct tensor ring decomposition for convolutional neural networks compressioncited by 0
  10. 2026Systematic Characterization of Minimal Deep Learning Architectures: A Unified Analysis of Convergence, Pruning, and Quantizationcited by 0