# Macro-block dropout for improved regularization in training end-to-end speech recognition models

arXiv: https://arxiv.org/abs/2212.14149 · HTML: https://mlsanity.com/paper/2212.14149/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 48 references that are in the map.
- First posted on arXiv 29 Dec 2022.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [DropBlock: A regularization method for convolutional networks](https://mlsanity.com/paper/1810.12890/) (2018)
- [FractalNet: Ultra-Deep Neural Networks without Residuals](https://mlsanity.com/paper/1605.07648/) (2016)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [Minimum Word Error Rate Training for Attention-based Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01818/) (2017)
- [ShakeDrop Regularization for Deep Residual Learning](https://mlsanity.com/paper/1802.02375/) (2018)
- [Delving into VoxCeleb: environment invariant speaker recognition](https://mlsanity.com/paper/1910.11238/) (2019)
- [A review of on-device fully neural end-to-end automatic speech recognition algorithms](https://mlsanity.com/paper/2012.07974/) (2020)
- [Efficient Implementation of the Room Simulator for Training Deep Neural Network Acoustic Models](https://mlsanity.com/paper/1712.03439/) (2017)
- [end-to-end training of a large vocabulary end-to-end speech recognition system](https://mlsanity.com/paper/1912.11040/) (2019)
- [Domain Mismatch Robust Acoustic Scene Classification using Channel Information Conversion](https://mlsanity.com/paper/1812.01731/) (2018)
- [Streaming end-to-end speech recognition with jointly trained neural feature enhancement](https://mlsanity.com/paper/2105.01254/) (2021)
- [power-law nonlinearity with maximally uniform distribution criterion for improved neural network training in automatic speech recognition](https://mlsanity.com/paper/1912.11041/) (2019)
- [Small energy masking for improved neural network training for end-to-end speech recognition](https://mlsanity.com/paper/2002.06312/) (2020)

## Abstract (arXiv)

This paper proposes a new regularization algorithm referred to as macro-block dropout. The overfitting issue has been a difficult problem in training large neural network models. The dropout technique has proven to be simple yet very effective for regularization by preventing complex co-adaptations during training. In our work, we define a macro-block that contains a large number of units from the input to a Recurrent Neural Network (RNN). Rather than applying dropout to each unit, we apply random dropout to each macro-block. This algorithm has the effect of applying different drop out rates for each layer even if we keep a constant average dropout rate, which has better regularization effects. In our experiments using Recurrent Neural Network-Transducer (RNN-T), this algorithm shows relatively 4.30 % and 6.13 % Word Error Rates (WERs) improvement over the conventional dropout on LibriSpeech test-clean and test-other. With an Attention-based Encoder-Decoder (AED) model, this algorithm shows relatively 4.36 % and 5.85 % WERs improvement over the conventional dropout on the same test sets.