mlsanityData to 7 Oct 2026

Real-Time Recurrent Learning using Trace Units in Reinforcement Learning

arXiv 2409.01449 ↗First posted 2 Sep 2024cs.LG cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Recurrent Neural Networks (RNNs) are used to learn representations in partially observable environments. For agents that learn online and continually interact with the environment, it is desirable to train RNNs with real-time recurrent learning (RTRL); unfortunately, RTRL is prohibitively expensive for standard RNNs. A promising direction is to use linear recurrent architectures (LRUs), where dense recurrent weights are replaced with a complex-valued diagonal, making RTRL efficient. In this work, we build on these insights to provide a lightweight but effective approach for training RNNs in online RL. We introduce Recurrent Trace Units (RTUs), a small modification on LRUs that we nonetheless find to have significant performance benefits over LRUs when trained with RTRL. We find RTUs significantly outperform other recurrent architectures across several partially observable environments while using significantly less computation.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 33 of its 48 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

33 of 48 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  7. 2015Automatic differentiation in machine learning: a surveycited by 846
  8. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  9. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  11. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  12. 2018Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNNcited by 142
21 more
  1. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 126
  2. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  3. 2022Discovered Policy Optimisationcited by 103
  4. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  5. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  6. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 59
  7. 2021Memory-based Deep Reinforcement Learning for POMDPscited by 53
  8. 2017Unbiased Online Recurrent Optimizationcited by 46
  9. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 46
  10. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45
  11. 2022A Modern Self-Referential Weight Matrix That Learns to Modify Itselfcited by 40
  12. 2024Towards Scalable and Stable Parallelization of Nonlinear RNNscited by 38
  13. 2019A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networkscited by 37
  14. 2023Online learning of long-range dependenciescited by 25
  15. 2023Exploring the Promise and Limits of Real-Time Recurrent Learningcited by 19
  16. 2015Recurrent Reinforcement Learning: A Hybrid Approachcited by 13
  17. 2020From Eye-blinks to State Construction: Diagnostic Benchmarks for Online Representation Learningcited by 13
  18. 2015Training recurrent networks online without backtrackingcited by 10
  19. 2024Rethinking Transformers in Solving POMDPscited by 9
  20. 2019On the Variance of Unbiased Online Recurrent Optimizationcited by 7
  21. 2023Time-Efficient Reinforcement Learning with Stochastic Stateful Policiescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 7.

10 papers.

  1. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  2. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 6
  3. 2025Investigating Memory in Model-Free RL with POPGym Arcadecited by 1
  4. 2026Value Bonuses using Ensemble Errors for Exploration in Reinforcement Learningcited by 1
  5. 2026Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic Viewcited by 1
  6. 2026Streaming Reinforcement Learning under Partial Observability with Real-Time Recurrent Learningcited by 1
  7. 2026Forager: a lightweight testbed for continual learning with partial observability in RLcited by 0
  8. 2026Foundations of Reinforcement Learning and Control:Connections and New Perspectivescited by 0
  9. 2026Reinforcement Learning with Complex (valued) Memoriescited by 0
  10. 2026Shared Phase and Retention Control for Efficient Adaptive Spectral Recurrencecited by 0