# Efficient batchwise dropout training using submatrices

arXiv: https://arxiv.org/abs/1502.02478 · HTML: https://mlsanity.com/paper/1502.02478/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Feb 2015.

## Built on by (most cited first, 3 of 3)

- [DISTREAL: Distributed Resource-Aware Learning in Heterogeneous Systems](https://mlsanity.com/paper/2112.08761/) (2021)
- [Concurrent Training and Layer Pruning of Deep Neural Networks](https://mlsanity.com/paper/2406.04549/) (2024)
- [Robust Learning of Parsimonious Deep Neural Networks](https://mlsanity.com/paper/2205.04650/) (2022)

## Abstract (arXiv)

Dropout is a popular technique for regularizing artificial neural networks. Dropout networks are generally trained by minibatch gradient descent with a dropout mask turning off some of the units---a different pattern of dropout is applied to every sample in the minibatch. We explore a very simple alternative to the dropout mask. Instead of masking dropped out units by setting them to zero, we perform matrix multiplication using a submatrix of the weight matrix---unneeded hidden units are never calculated. Performing dropout batchwise, so that one pattern of dropout is used for each sample in a minibatch, we can substantially reduce training times. Batchwise dropout can be used with fully-connected and convolutional neural networks.