# Exploring the Promise and Limits of Real-Time Recurrent Learning

arXiv: https://arxiv.org/abs/2305.19044 · HTML: https://mlsanity.com/paper/2305.19044/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 64 references that are in the map.
- First posted on arXiv 30 May 2023.

## Builds on (most cited first, 28 of 28)

- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN](https://mlsanity.com/paper/1803.04831/) (2018)
- [Multi-task Deep Reinforcement Learning with PopArt](https://mlsanity.com/paper/1809.04474/) (2018)
- [Going Beyond Linear Transformers with Recurrent Fast Weight Programmers](https://mlsanity.com/paper/2106.06295/) (2021)
- [Quasi-Recurrent Neural Networks](https://mlsanity.com/paper/1611.01576/) (2016)
- [Towards Interpretable Reinforcement Learning Using Attention Augmented Agents](https://mlsanity.com/paper/1906.02500/) (2019)
- [A Formal Hierarchy of RNN Architectures](https://mlsanity.com/paper/2004.08500/) (2020)
- [Simple Recurrent Units for Highly Parallelizable Recurrence](https://mlsanity.com/paper/1709.02755/) (2017)
- [Dynamic Evaluation of Neural Sequence Models](https://mlsanity.com/paper/1709.07432/) (2017)
- [Unbiased Online Recurrent Optimization](https://mlsanity.com/paper/1702.05043/) (2017)
- [TorchBeast: A PyTorch Platform for Distributed RL](https://mlsanity.com/paper/1910.03552/) (2019)
- [History Compression via Language Models in Reinforcement Learning](https://mlsanity.com/paper/2205.12258/) (2022)
- [Approximating Real-Time Recurrent Learning with Random Kronecker Factors](https://mlsanity.com/paper/1805.10842/) (2018)
- [Online learning of long-range dependencies](https://mlsanity.com/paper/2305.15947/) (2023)
- [Optimal Kronecker-Sum Approximation of Real Time Recurrent Learning](https://mlsanity.com/paper/1902.03993/) (2019)
- [Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets](https://mlsanity.com/paper/1901.09049/) (2019)
- [Scalable Real-Time Recurrent Learning Using Columnar-Constructive Networks](https://mlsanity.com/paper/2302.05326/) (2023)
- [IndyLSTMs: Independently Recurrent LSTMs](https://mlsanity.com/paper/1903.08023/) (2019)
- [Scalable Online Recurrent Learning Using Columnar Neural Networks](https://mlsanity.com/paper/2103.05787/) (2021)

## Built on by (most cited first, 19 of 19)

- [Streaming Deep Reinforcement Learning Finally Works](https://mlsanity.com/paper/2410.14606/) (2024)
- [Practical Computational Power of Linear Transformers and Their Recurrent and Self-Referential Extensions](https://mlsanity.com/paper/2310.16076/) (2023)
- [On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Era](https://mlsanity.com/paper/2402.08132/) (2024)
- [Real-Time Recurrent Learning using Trace Units in Reinforcement Learning](https://mlsanity.com/paper/2409.01449/) (2024)
- [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025)
- [Real-Time Recurrent Reinforcement Learning](https://mlsanity.com/paper/2311.04830/) (2023)
- [Recurrent Reinforcement Learning with Memoroids](https://mlsanity.com/paper/2402.09900/) (2024)
- [Estimating Post-Synaptic Effects for Online Training of Feed-Forward SNNs](https://mlsanity.com/paper/2311.16151/) (2023)
- [A Unified Framework for Neural Computation and Learning Over Time](https://mlsanity.com/paper/2409.12038/) (2024)
- [Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learning](https://mlsanity.com/paper/2605.24709/) (2026)
- [Nature-Inspired Local Propagation](https://mlsanity.com/paper/2402.05959/) (2024)
- [ODEStream: A Buffer-Free Online Learning Framework with ODE-based Adaptor for Streaming Time Series Forecasting](https://mlsanity.com/paper/2411.07413/) (2024)
- [Recurrent Joint Embedding Predictive Architecture with Recurrent Forward Propagation Learning](https://mlsanity.com/paper/2411.16695/) (2024)
- [Convergence Analysis of Real-time Recurrent Learning (RTRL) for a class of Recurrent Neural Networks](https://mlsanity.com/paper/2501.08040/) (2025)
- [A Scalable Hybrid Training Approach for Recurrent Spiking Neural Networks](https://mlsanity.com/paper/2506.14464/) (2025)
- [Who invented deep residual learning?](https://mlsanity.com/paper/2509.24732/) (2025)
- [Generalising E-prop to Deep Networks](https://mlsanity.com/paper/2512.24506/) (2025)
- [Learning Long-Range Dependencies with Temporal Predictive Coding](https://mlsanity.com/paper/2602.18131/) (2026)
- [Massive Redundancy in Gradient Transport Enables Sparse Online Learning](https://mlsanity.com/paper/2603.15195/) (2026)

## Abstract (arXiv)

Real-time recurrent learning (RTRL) for sequence-processing recurrent neural networks (RNNs) offers certain conceptual advantages over backpropagation through time (BPTT). RTRL requires neither caching past activations nor truncating context, and enables online learning. However, RTRL's time and space complexity make it impractical. To overcome this problem, most recent work on RTRL focuses on approximation theories, while experiments are often limited to diagnostic settings. Here we explore the practical promise of RTRL in more realistic settings. We study actor-critic methods that combine RTRL and policy gradients, and test them in several subsets of DMLab-30, ProcGen, and Atari-2600 environments. On DMLab memory tasks, our system trained on fewer than 1.2 B environmental frames is competitive with or outperforms well-known IMPALA and R2D2 baselines trained on 10 B frames. To scale to such challenging tasks, we focus on certain well-known neural architectures with element-wise recurrence, allowing for tractable RTRL without approximation. Importantly, we also discuss rarely addressed limitations of RTRL in real-world applications, such as its complexity in the multi-layer case.