mlsanityData to 7 Oct 2026

Magnitude Attention-based Dynamic Pruning

arXiv 2306.05056 ↗First posted 8 Jun 2023cs.CV cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Existing pruning methods utilize the importance of each weight based on specified criteria only when searching for a sparse structure but do not utilize it during training. In this work, we propose a novel approach - Magnitude Attention-based Dynamic Pruning (MAP) method, which applies the importance of weights throughout both the forward and backward paths to explore sparse model structures dynamically. Magnitude attention is defined based on the magnitude of weights as continuous real-valued numbers enabling a seamless transition from a redundant to an effective sparse network by promoting efficient exploration. Additionally, the attention mechanism ensures more effective updates for important layers within the sparse network. In later stages of training, our approach shifts from exploration to exploitation, exclusively updating the sparse model composed of crucial weights based on the explored structure, resulting in pruned models that not only achieve performance comparable to dense models but also outperform previous pruning methods on CIFAR-10/100 and ImageNet.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

32 of 42 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  7. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  8. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  9. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 467
  10. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 325
  11. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 238
  12. 2016Dynamic Network Surgery for Efficient DNNscited by 233
20 more
  1. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 210
  2. 2019Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterizationcited by 187
  3. 2020Soft Threshold Weight Reparameterization for Learnable Sparsitycited by 150
  4. 2017Deep Rewiring: Training very sparse deep networkscited by 136
  5. 2020Dynamic Model Pruning with Feedbackcited by 125
  6. 2021Sparse Training via Boosting Pruning Plasticity with Neuroregenerationcited by 83
  7. 2019Global Sparse Momentum SGD for Pruning Very Deep Neural Networkscited by 70
  8. 2016Learning the Number of Neurons in Deep Networkscited by 69
  9. 2020Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layerscited by 66
  10. 2019Discovering Neural Wiringscited by 56
  11. 2021Effective Sparsification of Neural Networks with Global Sparsity Constraintcited by 45
  12. 2022Prospect Pruning: Finding Trainable Weights at Initialization using Meta-Gradientscited by 34
  13. 2020Robust Pruning at Initializationcited by 31
  14. 2021Network Pruning That Matters: A Case Study on Retraining Variantscited by 25
  15. 2022Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruningcited by 13
  16. 2021How I Learned to Stop Worrying and Love Retrainingcited by 12
  17. 2023A Unified Framework for Soft Threshold Pruningcited by 12
  18. 2022Cyclical Pruning for Sparse Neural Networkscited by 6
  19. 2021Dynamic Collective Intelligence Learning: Finding Efficient Sparse Model via Refined Gradients for Pruned Weightscited by 4
  20. 2022Pushing the Efficiency Limit Using Structured Sparse Convolutionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 2; 2026: 1.

5 papers.

  1. 2024Density Adaptive Attention is All You Need: Robust Parameter-Efficient Fine-Tuning Across Multiple Modalitiescited by 2
  2. 2025SAP: Syntactic Attention Pruning for Transformer-based Language Modelscited by 1
  3. 2023MaxQ: Multi-Axis Query for N:M Sparsity Networkcited by 0
  4. 2025INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Modelscited by 0
  5. 2026SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Featurescited by 0