mlsanityData to 7 Oct 2026

Gated Recurrent Neural Networks with Weighted Time-Delay Feedback

arXiv 2212.00228 ↗First posted 1 Dec 2022cs.LG cs.NE stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

In this paper, we present a novel approach to modeling long-term dependencies in sequential data by introducing a gated recurrent unit (GRU) with a weighted time-delay feedback mechanism. Our proposed model, named τ-GRU, is a discretized version of a continuous-time formulation of a recurrent unit, where the dynamics are governed by delay differential equations (DDEs). We prove the existence and uniqueness of solutions for the continuous-time model and show that the proposed feedback mechanism can significantly improve the modeling of long-term dependencies. Our empirical results indicate that τ-GRU outperforms state-of-the-art recurrent units and gated recurrent architectures on a range of tasks, achieving faster convergence and better generalization.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 99 references that are in the map. First posted on arXiv 1 Dec 2022.

Builds on

37 of 99 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Neural Ordinary Differential Equationscited by 3,733
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  5. 2020Efficient Transformers: A Surveycited by 784
  6. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  7. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  8. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  9. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  10. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  11. 2015Unitary Evolution Recurrent Neural Networkscited by 291
  12. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 239
25 more
  1. 2022Liquid Structural State-Space Modelscited by 134
  2. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 133
  3. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  4. 2020Forecasting Sequential Data using Consistent Koopman Autoencoderscited by 114
  5. 2016Full-Capacity Unitary Recurrent Neural Networkscited by 90
  6. 2020Lipschitz Recurrent Neural Networkscited by 88
  7. 2017Dilated Recurrent Neural Networkscited by 83
  8. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 82
  9. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 81
  10. 2019Trivializations for Gradient-Based Optimization on Manifoldscited by 78
  11. 2019Physics-informed Autoencoders for Lyapunov-stable Fluid Flow Predictioncited by 67
  12. 2021Closed-form Continuous-time Neural Modelscited by 65
  13. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  14. 2018Can recurrent neural networks warp time?cited by 53
  15. 2017Skip RNN: Learning to Skip State Updates in Recurrent Neural Networkscited by 49
  16. 2021Noisy Recurrent Neural Networkscited by 42
  17. 2021Heavy Ball Neural Ordinary Differential Equationscited by 42
  18. 2021Long Expressive Memory for Sequence Modelingcited by 41
  19. 2020Continuous-in-Depth Neural Networkscited by 40
  20. 2019FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Networkcited by 38
  21. 2019Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamicscited by 28
  22. 2021Noisy Feature Mixupcited by 27
  23. 2021Neural Delay Differential Equationscited by 26
  24. 2021Stateful ODE-Nets using Basis Function Expansionscited by 14
  25. 2021A Differential Geometry Perspective on Orthogonal Recurrent Modelscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalizationcited by 18
  2. 2024Tuning Frequency Bias of State Space Modelscited by 16
  3. 2023Delayed Memory Unit: Modelling Temporal Dependency Through Delay Gatecited by 4
  4. 2025A Deep State Space Model for Rainfall-Runoff Simulationscited by 3
  5. 2024WaveCastNet: Rapid Wavefield Forecasting for Earthquake Early Warning via Deep Sequence to Sequence Learningcited by 0