mlsanityData to 14 Oct 2026

Can Local Representation Alignment RNNs Solve Temporal Tasks?

arXiv 2504.13531 ↗First posted 18 Apr 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recurrent Neural Networks (RNNs) are commonly used for real-time processing, streaming data, and cases where the amount of training samples is limited. Backpropagation Through Time (BPTT) is the predominant algorithm for training RNNs; however, it is frequently criticized for being prone to exploding and vanishing gradients and being biologically implausible. In this paper, we present and evaluate a target propagation-based method for RNNs, which uses local updates and seeks to reduce the said instabilities. Having stable RNN models increases their practical use in a wide range of fields such as natural language processing, time-series forecasting, anomaly detection, control systems, and robotics. The proposed solution uses local representation alignment (LRA). We thoroughly analyze the performance of this method, experiment with normalization and different local error functions, and invalidate certain assumptions about the behavior of this type of learning. Namely, we demonstrate that despite the decomposition of the network into sub-graphs, the model still suffers from vanishing gradients. We also show that gradient clipping as proposed in LRA has little to no effect on network performance. This results in an LRA RNN model that is very difficult to train due to vanishing gradients. We address this by introducing gradient regularization in the direction of the update and demonstrate that this modification promotes gradient flow and meaningfully impacts convergence. We compare and discuss the performance of the algorithm, and we show that the regularized LRA RNN considerably outperforms the unregularized version on three landmark tasks: temporal order, 3-bit temporal order, and random permutation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 63 references that are in the map. First posted on arXiv 18 Apr 2025.

Builds on

17 of 63 references.

References · 17

  1. 2015Unitary Evolution Recurrent Neural Networkscited by 294
  2. 2016Direct Feedback Alignment Provides Learning in Deep Neural Networkscited by 240
  3. 2019Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directionscited by 172
  4. 2014Difference Target Propagationcited by 148
  5. 2015A Simple Way to Initialize Recurrent Networks of Rectified Linear Unitscited by 130
  6. 2018Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architecturescited by 101
  7. 2017On orthogonality and learning recurrent networks with long term dependenciescited by 83
  8. 2021On the difficulty of learning chaotic dynamics with RNNscited by 67
  9. 2021Advancing RNN Transducer Technology for Speech Recognitioncited by 44
  10. 2018Biologically Motivated Algorithms for Propagating Local Target Representationscited by 43
  11. 2016Recurrent Orthogonal Networks and Long-Memory Taskscited by 38
  12. 2023Brain-inspired learning in artificial neural networks: a reviewcited by 26
5 more
  1. 2019Principled Training of Neural Networks with Direct Feedback Alignmentcited by 23
  2. 2022Meta-Learning Biologically Plausible Plasticity Rules with Random Feedback Pathwayscited by 14
  3. 2018Conducting Credit Assignment by Aligning Local Representationscited by 12
  4. 2021Context- and Sequence-Aware Convolutional Recurrent Encoder for Neural Machine Translationcited by 2
  5. 2022On the biological plausibility of orthogonal initialisation for solving gradient instability in deep neural networkscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Stabilizing RNN Gradients through Pre-training
  2. 2023Resurrecting Recurrent Neural Networks for Long Sequences
  3. 2025Fast Training of Recurrent Neural Networks with Stationary State Feedbacks
  4. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Models
  5. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval
  6. 2026DREAMSTATE: Diffusing States and Parameters for Recurrent Large Language Models
  7. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the story
  8. 2025IRNN: Innovation-driven Recurrent Neural Network for Time-Series Data Modeling and Prediction