# Optimal Kronecker-Sum Approximation of Real Time Recurrent Learning

arXiv: https://arxiv.org/abs/1902.03993 · HTML: https://mlsanity.com/paper/1902.03993/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2019.

## Built on by (most cited first, 22 of 22)

- [Unbiased Gradient Estimation in Unrolled Computation Graphs with Persistent Evolution Strategies](https://mlsanity.com/paper/2112.13835/) (2021)
- [Gradient Descent on Neurons and its Link to Approximate Second-Order Optimization](https://mlsanity.com/paper/2201.12250/) (2022)
- [Online learning of long-range dependencies](https://mlsanity.com/paper/2305.15947/) (2023)
- [Exploring the Promise and Limits of Real-Time Recurrent Learning](https://mlsanity.com/paper/2305.19044/) (2023)
- [Amortized Proximal Optimization](https://mlsanity.com/paper/2203.00089/) (2022)
- [Slower is Better: Revisiting the Forgetting Mechanism in LSTM for Slower Information Decay](https://mlsanity.com/paper/2105.05944/) (2021)
- [Low-Variance Gradient Estimation in Unrolled Computation Graphs with ES-Single](https://mlsanity.com/paper/2304.11153/) (2023)
- [Variance-Reduced Gradient Estimation via Noise-Reuse in Online Evolution Strategies](https://mlsanity.com/paper/2304.12180/) (2023)
- [Prediction of the Position of External Markers Using a Recurrent Neural Network Trained With Unbiased Online Recurrent Optimization for Safe Lung Cancer Radiotherapy](https://mlsanity.com/paper/2106.01100/) (2021)
- [Estimating Post-Synaptic Effects for Online Training of Feed-Forward SNNs](https://mlsanity.com/paper/2311.16151/) (2023)
- [Real-time respiratory motion forecasting with online learning of recurrent neural networks for accurate targeting in externally guided radiotherapy](https://mlsanity.com/paper/2403.01607/) (2024)
- [Mind the truncation gap: challenges of learning on dynamic graphs with recurrent architectures](https://mlsanity.com/paper/2412.21046/) (2024)
- [On Unbiased Low-Rank Approximation with Minimum Distortion](https://mlsanity.com/paper/2505.09647/) (2025)
- [Weaver: Kronecker Product Approximations of Spatiotemporal Attention for Traffic Network Forecasting](https://mlsanity.com/paper/2511.08888/) (2025)
- [Three factor delay learning rules for spiking neural networks](https://mlsanity.com/paper/2601.00668/) (2026)
- [Frame forecasting in cine MRI using the PCA respiratory motion model: comparing recurrent neural networks trained online and transformers](https://mlsanity.com/paper/2410.05882/) (2024)
- [Convergence Analysis of Real-time Recurrent Learning (RTRL) for a class of Recurrent Neural Networks](https://mlsanity.com/paper/2501.08040/) (2025)
- [Generalising E-prop to Deep Networks](https://mlsanity.com/paper/2512.24506/) (2025)
- [Unbiased Approximate Vector-Jacobian Products for Efficient Backpropagation](https://mlsanity.com/paper/2602.14701/) (2026)
- [Massive Redundancy in Gradient Transport Enables Sparse Online Learning](https://mlsanity.com/paper/2603.15195/) (2026)
- [Immediate Derivatives Suffice for Online Recurrent Adaptation](https://mlsanity.com/paper/2603.28750/) (2026)
- [Large Distant Gradients Need Not Be Reliable: reliability-weighted credit assignment for long-horizon autoregressive forecasting](https://mlsanity.com/paper/2609.12890/) (2026)

## Abstract (arXiv)

One of the central goals of Recurrent Neural Networks (RNNs) is to learn long-term dependencies in sequential data. Nevertheless, the most popular training method, Truncated Backpropagation through Time (TBPTT), categorically forbids learning dependencies beyond the truncation horizon. In contrast, the online training algorithm Real Time Recurrent Learning (RTRL) provides untruncated gradients, with the disadvantage of impractically large computational costs. Recently published approaches reduce these costs by providing noisy approximations of RTRL. We present a new approximation algorithm of RTRL, Optimal Kronecker-Sum Approximation (OK). We prove that OK is optimal for a class of approximations of RTRL, which includes all approaches published so far. Additionally, we show that OK has empirically negligible noise: Unlike previous algorithms it matches TBPTT in a real world task (character-level Penn TreeBank) and can exploit online parameter updates to outperform TBPTT in a synthetic string memorization task. Code availiable on github.