mlsanityData to 14 Oct 2026

Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gate

arXiv 2310.14982 ↗First posted 23 Oct 2023cs.NE cs.LG eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

Recurrent Neural Networks (RNNs) are widely recognized for their proficiency in modeling temporal dependencies, making them highly prevalent in sequential data processing applications. Nevertheless, vanilla RNNs are confronted with the well-known issue of gradient vanishing and exploding, posing a significant challenge for learning and establishing long-range dependencies. Additionally, gated RNNs tend to be over-parameterized, resulting in poor computational efficiency and network generalization. To address these challenges, this paper proposes a novel Delayed Memory Unit (DMU). The DMU incorporates a delay line structure along with delay gates into vanilla RNN, thereby enhancing temporal interaction and facilitating temporal credit assignment. Specifically, the DMU is designed to directly distribute the input information to the optimal time instant in the future, rather than aggregating and redistributing it over time through intricate network dynamics. Our proposed DMU demonstrates superior temporal modeling capabilities across a broad range of sequential modeling tasks, utilizing considerably fewer parameters than other state-of-the-art gated RNN models in applications such as speech recognition, radar gesture recognition, ECG waveform segmentation, and permuted sequential image classification.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 31 of its 57 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

31 of 57 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  6. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  7. 2018SLAYER: Spike Layer Error Reassignment in Timecited by 266
  8. 2022Neural Networks and the Chomsky Hierarchycited by 217
  9. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 143
  10. 2019The Heidelberg spiking datasets for the systematic evaluation of spiking neural networkscited by 137
  11. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  12. 2021Accurate and efficient time-domain classification with adaptive spiking recurrent neural networkscited by 115
19 more
  1. 2014Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognitioncited by 111
  2. 2021Temporal-wise Attention Spiking Neural Networks for Event Streams Classificationcited by 106
  3. 2020Lipschitz Recurrent Neural Networkscited by 89
  4. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 82
  5. 2023Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacingscited by 50
  6. 2018Light Gated Recurrent Units for Speech Recognitioncited by 48
  7. 2020Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networkscited by 43
  8. 2021Long Expressive Memory for Sequence Modelingcited by 41
  9. 2018The PyTorch-Kaldi Speech Recognition Toolkitcited by 40
  10. 2020A Comparison of LSTM and BERT for Small Corpuscited by 40
  11. 2018Efficient keyword spotting using dilated convolutions and gatingcited by 21
  12. 2019Towards Non-saturating Recurrent Units for Modelling Long-term Dependenciescited by 15
  13. 2020Multi-timescale Representation Learning in LSTM Language Modelscited by 13
  14. 2022Axonal Delay As a Short-Term Memory for Feed Forward Deep Spiking Neural Networkscited by 11
  15. 2023Adaptive Axonal Delays in feedforward spiking neural networks for accurate spoken word recognitioncited by 11
  16. 2020Markovian RNN: An Adaptive Time Series Prediction Network with HMM-based Switching for Nonstationary Environmentscited by 9
  17. 2022Gated Recurrent Neural Networks with Weighted Time-Delay Feedbackcited by 5
  18. 2017Improving speech recognition by revising gated recurrent unitscited by 3
  19. 2013Delay Learning Architectures for Memory and Classificationcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2025Beyond Rate Coding: Surrogate Gradients Enable Spike Timing Learning in Spiking Neural Networkscited by 3
  2. 2025Towards Efficient and Accurate Spiking Neural Networks via Adaptive Bit Allocationcited by 2
  3. 2025Exploiting heterogeneous delays for efficient computation in low-bit neural networkscited by 2
  4. 2025SpikCommander: A High-performance Spiking Transformer with Multi-view Learning for Efficient Speech Command Recognitioncited by 2
  5. 2024EEG decoding with conditional identification informationcited by 1