# Full deep neural network training on a pruned weight budget

arXiv: https://arxiv.org/abs/1806.06949 · HTML: https://mlsanity.com/paper/1806.06949/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2018.

## Built on by (most cited first, 11 of 11)

- [Efficient Fine-Tuning of BERT Models on the Edge](https://mlsanity.com/paper/2205.01541/) (2022)
- [SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks](https://mlsanity.com/paper/2302.04852/) (2023)
- [Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs](https://mlsanity.com/paper/2107.13490/) (2021)
- [Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining](https://mlsanity.com/paper/2407.19126/) (2024)
- [A Generalized Lottery Ticket Hypothesis](https://mlsanity.com/paper/2107.06825/) (2021)
- [Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models](https://mlsanity.com/paper/2304.13718/) (2023)
- [Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection](https://mlsanity.com/paper/2310.13183/) (2023)
- [LearningGroup: A Real-Time Sparse Training on FPGA via Learnable Weight Grouping for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2210.16624/) (2022)
- [Competitive plasticity to reduce the energetic costs of learning](https://mlsanity.com/paper/2304.02594/) (2023)
- [Dynamic Neural Network Architectural and Topological Adaptation and Related Methods -- A Survey](https://mlsanity.com/paper/2108.10066/) (2021)
- [The Multiple Subnetwork Hypothesis: Enabling Multidomain Learning by Isolating Task-Specific Subnetworks in Feedforward Neural Networks](https://mlsanity.com/paper/2207.08821/) (2022)

## Abstract (arXiv)

We introduce a DNN training technique that learns only a fraction of the full parameter set without incurring an accuracy penalty. To do this, our algorithm constrains the total number of weights updated during backpropagation to those with the highest total gradients. The remaining weights are not tracked, and their initial value is regenerated at every access to avoid storing them in memory. This can dramatically reduce the number of off-chip memory accesses during both training and inference, a key component of the energy needs of DNN accelerators. By ensuring that the total weight diffusion remains close to that of baseline unpruned SGD, networks pruned using our technique are able to retain state-of-the-art accuracy across network architectures -- including networks previously identified as difficult to compress, such as Densenet and WRN. With ResNet18 on ImageNet, we observe an 11.7× weight reduction with no accuracy loss, and up to 24.4× with a small accuracy impact.