# Light Gated Recurrent Units for Speech Recognition

arXiv: https://arxiv.org/abs/1803.10225 · HTML: https://mlsanity.com/paper/1803.10225/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Mar 2018.

## Built on by (most cited first, 30 of 48)

- [SpeechBrain: A General-Purpose Speech Toolkit](https://mlsanity.com/paper/2106.04624/) (2021)
- [Unsupervised Speech Recognition](https://mlsanity.com/paper/2105.11084/) (2021)
- [Open-Source Conversational AI with SpeechBrain 1.0](https://mlsanity.com/paper/2407.00463/) (2024)
- [SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts](https://mlsanity.com/paper/2105.03036/) (2021)
- [An Experimental Review on Deep Learning Architectures for Time Series Forecasting](https://mlsanity.com/paper/2103.12057/) (2021)
- [DANSE: Data-driven Non-linear State Estimation of Model-free Process in Unsupervised Learning Setup](https://mlsanity.com/paper/2306.03897/) (2023)
- [OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction](https://mlsanity.com/paper/2512.16842/) (2025)
- [TinyML for Ubiquitous Edge AI](https://mlsanity.com/paper/2102.01255/) (2021)
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021)
- [Understand me, if you refer to Aspect Knowledge: Knowledge-aware Gated Recurrent Memory Network](https://mlsanity.com/paper/2108.02352/) (2021)
- [Synthesizing Dysarthric Speech Using Multi-talker TTS for Dysarthric Speech Recognition](https://mlsanity.com/paper/2201.11571/) (2022)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [Effective Low-Cost Time-Domain Audio Separation Using Globally Attentive Locally Recurrent Networks](https://mlsanity.com/paper/2101.05014/) (2021)
- [Accurate synthesis of Dysarthric Speech for ASR data augmentation](https://mlsanity.com/paper/2308.08438/) (2023)
- [Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate](https://mlsanity.com/paper/2310.14982/) (2023)
- [Robust Classification using Hidden Markov Models and Mixtures of Normalizing Flows](https://mlsanity.com/paper/2102.07284/) (2021)
- [Normalizing Flow based Hidden Markov Models for Classification of Speech Phones with Explainability](https://mlsanity.com/paper/2107.00730/) (2021)
- [Continuous Speech for Improved Learning Pathological Voice Disorders](https://mlsanity.com/paper/2202.10777/) (2022)
- [FedorAS: Federated Architecture Search under system heterogeneity](https://mlsanity.com/paper/2206.11239/) (2022)
- [Learning to Stop Overthinking at Test Time](https://mlsanity.com/paper/2502.10954/) (2025)
- [Conditional independence for pretext task selection in Self-supervised speech representation learning](https://mlsanity.com/paper/2104.07388/) (2021)
- [Sequence Prediction Under Missing Data : An RNN Approach Without Imputation](https://mlsanity.com/paper/2208.08933/) (2022)
- [Stabilising and accelerating light gated recurrent units for automatic speech recognition](https://mlsanity.com/paper/2302.10144/) (2023)
- [Quran Recitation Recognition using End-to-End Deep Learning](https://mlsanity.com/paper/2305.07034/) (2023)
- [Automatic speech recognition for the Nepali language using CNN, bidirectional LSTM and ResNet](https://mlsanity.com/paper/2406.17825/) (2024)
- [Tune-In: Training Under Negative Environments with Interference for Attention Networks Simulating Cocktail Party Effect](https://mlsanity.com/paper/2103.01461/) (2021)
- [Trainability, Expressivity and Interpretability in Gated Neural ODEs](https://mlsanity.com/paper/2307.06398/) (2023)
- [Frameless Graph Knowledge Distillation](https://mlsanity.com/paper/2307.06631/) (2023)
- [Hierarchical Recurrent Adapters for Efficient Multi-Task Adaptation of Large Speech Models](https://mlsanity.com/paper/2403.19709/) (2024)
- [Gates Are Not What You Need in RNNs](https://mlsanity.com/paper/2108.00527/) (2021)

## Abstract (arXiv)

A field that has directly benefited from the recent advances in deep learning is Automatic Speech Recognition (ASR). Despite the great achievements of the past decades, however, a natural and robust human-machine speech interaction still appears to be out of reach, especially in challenging environments characterized by significant noise and reverberation. To improve robustness, modern speech recognizers often employ acoustic models based on Recurrent Neural Networks (RNNs), that are naturally able to exploit large time contexts and long-term speech modulations. It is thus of great interest to continue the study of proper techniques for improving the effectiveness of RNNs in processing speech signals. In this paper, we revise one of the most popular RNN models, namely Gated Recurrent Units (GRUs), and propose a simplified architecture that turned out to be very effective for ASR. The contribution of this work is two-fold: First, we analyze the role played by the reset gate, showing that a significant redundancy with the update gate occurs. As a result, we propose to remove the former from the GRU design, leading to a more efficient and compact single-gate model. Second, we propose to replace hyperbolic tangent with ReLU activations. This variation couples well with batch normalization and could help the model learn long-term dependencies without numerical issues. Results show that the proposed architecture, called Light GRU (Li-GRU), not only reduces the per-epoch training time by more than 30% over a standard GRU, but also consistently improves the recognition accuracy across different tasks, input features, noisy conditions, as well as across different ASR paradigms, ranging from standard DNN-HMM speech recognizers to end-to-end CTC models.