# Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks

arXiv: https://arxiv.org/abs/1812.08119 · HTML: https://mlsanity.com/paper/1812.08119/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Dec 2018.

## Built on by (most cited first, 3 of 3)

- [Locally Sparse Neural Networks for Tabular Biomedical Data](https://mlsanity.com/paper/2106.06468/) (2021)
- [The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks](https://mlsanity.com/paper/2104.06153/) (2021)
- [Spiking Synaptic Penalty: Appropriate Penalty Term for Energy-Efficient Spiking Neural Networks](https://mlsanity.com/paper/2302.01500/) (2023)

## Abstract (arXiv)

In recent years, deep neural networks (DNNs) have been applied to various machine leaning tasks, including image recognition, speech recognition, and machine translation. However, large DNN models are needed to achieve state-of-the-art performance, exceeding the capabilities of edge devices. Model reduction is thus needed for practical use. In this paper, we point out that deep learning automatically induces group sparsity of weights, in which all weights connected to an output channel (node) are zero, when training DNNs under the following three conditions: (1) rectified-linear-unit (ReLU) activations, (2) an L2-regularized objective function, and (3) the Adam optimizer. Next, we analyze this behavior both theoretically and experimentally, and propose a simple model reduction method: eliminate the zero weights after training the DNN. In experiments on MNIST and CIFAR-10 datasets, we demonstrate the sparsity with various training setups. Finally, we show that our method can efficiently reduce the model size and performs well relative to methods that use a sparsity-inducing regularizer.