# Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization

arXiv: https://arxiv.org/abs/1812.00481 · HTML: https://mlsanity.com/paper/1812.00481/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2018.

## Built on by (most cited first, 10 of 10)

- [Diagnosing and Re-learning for Balanced Multimodal Learning](https://mlsanity.com/paper/2407.09705/) (2024)
- [Knowledge Evolution in Neural Networks](https://mlsanity.com/paper/2103.05152/) (2021)
- [Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation](https://mlsanity.com/paper/2104.11939/) (2021)
- [Network Pruning via Resource Reallocation](https://mlsanity.com/paper/2103.01847/) (2021)
- [The Importance of Being Parameters: An Intra-Distillation Method for Serious Gains](https://mlsanity.com/paper/2205.11416/) (2022)
- [Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization](https://mlsanity.com/paper/2109.11045/) (2021)
- [DistFL: Distribution-aware Federated Learning for Mobile Scenarios](https://mlsanity.com/paper/2110.11619/) (2021)
- [Synaptic Stripping: How Pruning Can Bring Dead Neurons Back To Life](https://mlsanity.com/paper/2302.05818/) (2023)
- [Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization](https://mlsanity.com/paper/2102.06870/) (2021)
- [Weight Evolution: Improving Deep Neural Networks Training through Evolving Inferior Weight Values](https://mlsanity.com/paper/2110.04492/) (2021)

## Abstract (arXiv)

In this paper, we study the problem of improving computational resource utilization of neural networks. Deep neural networks are usually over-parameterized for their tasks in order to achieve good performances, thus are likely to have underutilized computational resources. This observation motivates a lot of research topics, e.g. network pruning, architecture search, etc. As models with higher computational costs (e.g. more parameters or more computations) usually have better performances, we study the problem of improving the resource utilization of neural networks so that their potentials can be further realized. To this end, we propose a novel optimization method named Neural Rejuvenation. As its name suggests, our method detects dead neurons and computes resource utilization in real time, rejuvenates dead neurons by resource reallocation and reinitialization, and trains them with new training schemes. By simply replacing standard optimizers with Neural Rejuvenation, we are able to improve the performances of neural networks by a very large margin while using similar training efforts and maintaining their original resource usages.