mlsanityData to 7 Oct 2026

Online learning of long-range dependencies

arXiv 2305.15947 ↗First posted 25 May 2023cs.LG cs.NEcited by 25

Not on the map. Why?

Abstract (arXiv)

Online learning holds the promise of enabling efficient long-term credit assignment in recurrent neural networks. However, current algorithms fall short of offline backpropagation by either not being scalable or failing to learn long-range dependencies. Here we present a high-performance online learning algorithm that merely doubles the memory and computational requirements of a single inference pass. We achieve this by leveraging independent recurrent modules in multi-layer networks, an architectural motif that has recently been shown to be particularly powerful. Experiments on synthetic memory problems and on the challenging long-range arena benchmark suite reveal that our algorithm performs competitively, establishing a new standard for what can be achieved through online learning. This ability to learn long-range dependencies offers a new perspective on learning in the brain and opens a promising avenue in neuromorphic computing.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 28 of its 66 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

28 of 66 references.

References · 28

  1. 2016Layer Normalizationcited by 5,579
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  5. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  7. 2016Language Modeling with Gated Convolutional Networkscited by 688
  8. 2014Neural Turing Machinescited by 630
  9. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  10. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  11. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  12. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
16 more
  1. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 251
  2. 2014Difference Target Propagationcited by 144
  3. 2016Decoupled Neural Interfaces using Synthetic Gradientscited by 135
  4. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  5. 2022Gradients without Backpropagationcited by 73
  6. 2017Unbiased Online Recurrent Optimizationcited by 46
  7. 2022Holomorphic Equilibrium Propagation Computes Exact Gradients Through Finite Size Oscillationscited by 40
  8. 2019A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networkscited by 37
  9. 2020Brain-Inspired Learning on Neuromorphic Substratescited by 35
  10. 2022ReckOn: A 28nm Sub-mm2 Task-Agnostic Spiking Recurrent Neural Network Processor Enabling On-Chip Learning over Second-Long Timescalescited by 35
  11. 2018Approximating Real-Time Recurrent Learning with Random Kronecker Factorscited by 29
  12. 2019Optimal Kronecker-Sum Approximation of Real Time Recurrent Learningcited by 21
  13. 2020A Practical Sparse Approximation for Real Time Recurrent Learningcited by 18
  14. 2022The least-control principle for local learning at equilibriumcited by 16
  15. 2023Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networkscited by 11
  16. 2021Online Training of Spiking Recurrent Neural Networks with Phase-Change Memory Synapsescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 10; 2025: 5; 2026: 5.

25 papers.

Sort:

By area:

  1. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  2. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  3. 2023Exploring the Promise and Limits of Real-Time Recurrent Learningcited by 19
  4. 2024On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Eracited by 17
  5. 2024Backpropagation through space, time, and the braincited by 10
  6. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learningcited by 10
  7. 2023Gated recurrent neural networks discover attentioncited by 7
  8. 2023Real-Time Recurrent Reinforcement Learningcited by 6
  9. 2025Learning long range dependencies through time reversal symmetry breakingcited by 6
  10. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  11. 2023AGaLiTe: Approximate Gated Linear Transformers for Online Reinforcement Learningcited by 5
  12. 2025Weight-Space Linear Recurrent Neural Networkscited by 5
13 more
  1. 2023Fading memory as inductive bias in residual recurrent networkscited by 2
  2. 2024Lifelong Learning of Video Diffusion Models From a Single Video Streamcited by 2
  3. 2024Real-time respiratory motion forecasting with online learning of recurrent neural networks for accurate targeting in externally guided radiotherapycited by 1
  4. 2024On the low-shot transferability of [V]-Mambacited by 1
  5. 2024A Unified Framework for Neural Computation and Learning Over Timecited by 1
  6. 2025Online Fine-Tuning of Carbon Emission Predictions using Real-Time Recurrent Learning for State Space Modelscited by 1
  7. 2024Nature-Inspired Local Propagationcited by 0
  8. 2025A Scalable Hybrid Training Approach for Recurrent Spiking Neural Networkscited by 0
  9. 2026Adaptive Control in Autonomous Driving via Real-Time Recurrent RLcited by 0
  10. 2026Learning Long-Range Dependencies with Temporal Predictive Codingcited by 0
  11. 2026Massive Redundancy in Gradient Transport Enables Sparse Online Learningcited by 0
  12. 2026Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networkscited by 0
  13. 2026Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrencecited by 0