# Ising-Dropout: A Regularization Method for Training and Compression of Deep Neural Networks

arXiv: https://arxiv.org/abs/1902.08673 · HTML: https://mlsanity.com/paper/1902.08673/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Feb 2019.

## Built on by (most cited first, 6 of 6)

- [S-Rocket: Selective Random Convolution Kernels for Time Series Classification](https://mlsanity.com/paper/2203.03445/) (2022)
- [A Survey on Dropout Methods and Experimental Verification in Recommendation](https://mlsanity.com/paper/2204.02027/) (2022)
- [Frequency Dropout: Feature-Level Regularization via Randomized Filtering](https://mlsanity.com/paper/2209.09844/) (2022)
- [Dropout Drops Double Descent](https://mlsanity.com/paper/2305.16179/) (2023)
- [Exploration of Various Deep Learning Models for Increased Accuracy in Automatic Polyp Detection](https://mlsanity.com/paper/2203.04093/) (2022)
- [MWaste: A Deep Learning Approach to Manage Household Waste](https://mlsanity.com/paper/2304.14498/) (2023)

## Abstract (arXiv)

Overfitting is a major problem in training machine learning models, specifically deep neural networks. This problem may be caused by imbalanced datasets and initialization of the model parameters, which conforms the model too closely to the training data and negatively affects the generalization performance of the model for unseen data. The original dropout is a regularization technique to drop hidden units randomly during training. In this paper, we propose an adaptive technique to wisely drop the visible and hidden units in a deep neural network using Ising energy of the network. The preliminary results show that the proposed approach can keep the classification performance competitive to the original network while eliminating optimization of unnecessary network parameters in each training cycle. The dropout state of units can also be applied to the trained (inference) model. This technique could compress the network in terms of number of parameters up to 41.18% and 55.86% for the classification task on the MNIST and Fashion-MNIST datasets, respectively.