# Energy-based Dropout in Restricted Boltzmann Machines: Why not go random

arXiv: https://arxiv.org/abs/2101.06741 · HTML: https://mlsanity.com/paper/2101.06741/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 28 references that are in the map.
- First posted on arXiv 17 Jan 2021.

## Builds on (most cited first, 5 of 5)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Deep Learning for Classical Japanese Literature](https://mlsanity.com/paper/1812.01718/) (2018)
- [Temperature-Based Deep Boltzmann Machines](https://mlsanity.com/paper/1608.07719/) (2016)
- [Learnergy: Energy-based Machine Learners](https://mlsanity.com/paper/2003.07443/) (2020)
- [Regularization for Unsupervised Deep Neural Nets](https://mlsanity.com/paper/1608.04426/) (2016)

## Built on by (most cited first, 2 of 2)

- [MaxDropoutV2: An Improved Method to Drop out Neurons in Convolutional Neural Networks](https://mlsanity.com/paper/2203.02740/) (2022)
- [Evolving Restricted Boltzmann Machine-Kohonen Network for Online Clustering](https://mlsanity.com/paper/2402.09167/) (2024)

## Abstract (arXiv)

Deep learning architectures have been widely fostered throughout the last years, being used in a wide range of applications, such as object recognition, image reconstruction, and signal processing. Nevertheless, such models suffer from a common problem known as overfitting, which limits the network from predicting unseen data effectively. Regularization approaches arise in an attempt to address such a shortcoming. Among them, one can refer to the well-known Dropout, which tackles the problem by randomly shutting down a set of neurons and their connections according to a certain probability. Therefore, this approach does not consider any additional knowledge to decide which units should be disconnected. In this paper, we propose an energy-based Dropout (E-Dropout) that makes conscious decisions whether a neuron should be dropped or not. Specifically, we design this regularization method by correlating neurons and the model's energy as an importance level for further applying it to energy-based models, such as Restricted Boltzmann Machines (RBMs). The experimental results over several benchmark datasets revealed the proposed approach's suitability compared to the traditional Dropout and the standard RBMs.