# Dynamic Model Pruning with Feedback

arXiv: https://arxiv.org/abs/2006.07253 · HTML: https://mlsanity.com/paper/2006.07253/

## Key facts

- Cited by 125 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2020.

## Built on by (most cited first, 30 of 125)

- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning](https://mlsanity.com/paper/2109.05687/) (2021)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Sparse Training via Boosting Pruning Plasticity with Neuroregeneration](https://mlsanity.com/paper/2106.10404/) (2021)
- [CHIP: CHannel Independence-based Pruning for Compact Neural Networks](https://mlsanity.com/paper/2110.13981/) (2021)
- [Compressing LLMs: The Truth is Rarely Pure and Never Simple](https://mlsanity.com/paper/2310.01382/) (2023)
- [AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks](https://mlsanity.com/paper/2106.12379/) (2021)
- [Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression](https://mlsanity.com/paper/2403.15447/) (2024)
- [How Well Do Sparse Imagenet Models Transfer?](https://mlsanity.com/paper/2111.13445/) (2021)
- [Scaling Laws for Sparsely-Connected Foundation Models](https://mlsanity.com/paper/2309.08520/) (2023)
- [Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models](https://mlsanity.com/paper/2210.05497/) (2022)
- [The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter](https://mlsanity.com/paper/2306.03805/) (2023)
- [Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition](https://mlsanity.com/paper/2107.11442/) (2021)
- [On the Existence of Universal Lottery Tickets](https://mlsanity.com/paper/2111.11146/) (2021)
- [GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization](https://mlsanity.com/paper/2109.02220/) (2021)
- [An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers](https://mlsanity.com/paper/2208.06118/) (2022)
- [NOLA: Compressing LoRA using Linear Combination of Random Basis](https://mlsanity.com/paper/2310.02556/) (2023)
- [Consistent Sparse Deep Learning: Theory and Computation](https://mlsanity.com/paper/2102.13229/) (2021)
- [ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training](https://mlsanity.com/paper/2110.05323/) (2021)
- [Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!](https://mlsanity.com/paper/2303.02141/) (2023)
- [Time-Correlated Sparsification for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2101.08837/) (2021)
- [Effective Model Sparsification by Scheduled Grow-and-Prune Methods](https://mlsanity.com/paper/2106.09857/) (2021)
- [Understanding the Dynamics of DNNs Using Graph Modularity](https://mlsanity.com/paper/2111.12485/) (2021)
- [Network Pruning That Matters: A Case Study on Retraining Variants](https://mlsanity.com/paper/2105.03193/) (2021)
- [Every Parameter Matters: Ensuring the Convergence of Federated Learning with Dynamic Heterogeneous Models Reduction](https://mlsanity.com/paper/2310.08670/) (2023)
- [The Elastic Lottery Ticket Hypothesis](https://mlsanity.com/paper/2103.16547/) (2021)
- [PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation](https://mlsanity.com/paper/2106.14681/) (2021)
- [Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions](https://mlsanity.com/paper/2209.08554/) (2022)
- [Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging](https://mlsanity.com/paper/2306.16788/) (2023)
- [RED : Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networks](https://mlsanity.com/paper/2105.14797/) (2021)

## Abstract (arXiv)

Deep neural networks often have millions of parameters. This can hinder their deployment to low-end devices, not only due to high memory requirements but also because of increased latency at inference. We propose a novel model compression method that generates a sparse trained model without additional overhead: by allowing (i) dynamic allocation of the sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve the performance). We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models. Moreover, their performance surpasses that of models generated by all previously proposed pruning schemes.