# Stabilising and accelerating light gated recurrent units for automatic speech recognition

arXiv: https://arxiv.org/abs/2302.10144 · HTML: https://mlsanity.com/paper/2302.10144/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 16 Feb 2023.

## Builds on (most cited first, 15 of 15)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Unitary Evolution Recurrent Neural Networks](https://mlsanity.com/paper/1511.06464/) (2015)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [A Comparative Study on Transformer vs RNN in Speech Applications](https://mlsanity.com/paper/1909.06317/) (2019)
- [On orthogonality and learning recurrent networks with long term dependencies](https://mlsanity.com/paper/1702.00071/) (2017)
- [Light Gated Recurrent Units for Speech Recognition](https://mlsanity.com/paper/1803.10225/) (2018)

## Built on by (most cited first, 3 of 3)

- [Open-Source Conversational AI with SpeechBrain 1.0](https://mlsanity.com/paper/2407.00463/) (2024)
- [Efficiency-oriented approaches for self-supervised speech representation learning](https://mlsanity.com/paper/2312.11142/) (2023)
- [Measuring the Redundancy of Decoder Layers in SpeechLLMs](https://mlsanity.com/paper/2603.05121/) (2026)

## Abstract (arXiv)

The light gated recurrent units (Li-GRU) is well-known for achieving impressive results in automatic speech recognition (ASR) tasks while being lighter and faster to train than a standard gated recurrent units (GRU). However, the unbounded nature of its rectified linear unit on the candidate recurrent gate induces an important gradient exploding phenomenon disrupting the training process and preventing it from being applied to famous datasets. In this paper, we theoretically and empirically derive the necessary conditions for its stability as well as engineering mechanisms to speed up by a factor of five its training time, hence introducing a novel version of this architecture named SLi-GRU. Then, we evaluate its performance both on a toy task illustrating its newly acquired capabilities and a set of three different ASR datasets demonstrating lower word error rates compared to more complex recurrent neural networks.