# Real-Time Recurrent Reinforcement Learning

arXiv: https://arxiv.org/abs/2311.04830 · HTML: https://mlsanity.com/paper/2311.04830/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Behaviour Suite for Reinforcement Learning](https://mlsanity.com/paper/1908.03568/) (2019)
- [Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPs](https://mlsanity.com/paper/2110.05038/) (2021)
- [Differentiable plasticity: training plastic neural networks with backpropagation](https://mlsanity.com/paper/1804.02464/) (2018)
- [Random feedback weights support learning in deep neural networks](https://mlsanity.com/paper/1411.0247/) (2014)
- [Unbiased Online Recurrent Optimization](https://mlsanity.com/paper/1702.05043/) (2017)
- [A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks](https://mlsanity.com/paper/1907.02649/) (2019)
- [Brain-Inspired Learning on Neuromorphic Substrates](https://mlsanity.com/paper/2010.11931/) (2020)
- [Approximating Real-Time Recurrent Learning with Random Kronecker Factors](https://mlsanity.com/paper/1805.10842/) (2018)
- [Online learning of long-range dependencies](https://mlsanity.com/paper/2305.15947/) (2023)
- [Exploring the Promise and Limits of Real-Time Recurrent Learning](https://mlsanity.com/paper/2305.19044/) (2023)
- [Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems](https://mlsanity.com/paper/2209.09174/) (2022)
- [Deep Reinforcement Learning with Modulated Hebbian plus Q Network Architecture](https://mlsanity.com/paper/1909.09902/) (2019)
- [Reconciling λ-Returns with Experience Replay](https://mlsanity.com/paper/1810.09967/) (2018)

## Built on by (most cited first, 6 of 6)

- [Brain-Inspired Machine Intelligence: A Survey of Neurobiologically-Plausible Credit Assignment](https://mlsanity.com/paper/2312.09257/) (2023)
- [Online Fine-Tuning of Carbon Emission Predictions using Real-Time Recurrent Learning for State Space Models](https://mlsanity.com/paper/2508.00804/) (2025)
- [Success in Humanoid Reinforcement Learning under Partial Observation](https://mlsanity.com/paper/2507.18883/) (2025)
- [TubeDAgger: Reducing the Number of Expert Interventions with Stochastic Reach-Tubes](https://mlsanity.com/paper/2510.00906/) (2025)
- [Adaptive Control in Autonomous Driving via Real-Time Recurrent RL](https://mlsanity.com/paper/2602.02236/) (2026)
- [Dynamical Priors as a Training Objective in Reinforcement Learning](https://mlsanity.com/paper/2604.21464/) (2026)

## Abstract (arXiv)

We introduce a biologically plausible RL framework for solving tasks in partially observable Markov decision processes (POMDPs). The proposed algorithm combines three integral parts: (1) A Meta-RL architecture, resembling the mammalian basal ganglia; (2) A biologically plausible reinforcement learning algorithm, exploiting temporal difference learning and eligibility traces to train the policy and the value-function; (3) An online automatic differentiation algorithm for computing the gradients with respect to parameters of a shared recurrent network backbone. Our experimental results show that the method is capable of solving a diverse set of partially observable reinforcement learning tasks. The algorithm we call real-time recurrent reinforcement learning (RTRRL) serves as a model of learning in biological neural networks, mimicking reward pathways in the basal ganglia.