# Immediate Derivatives Suffice for Online Recurrent Adaptation

arXiv: https://arxiv.org/abs/2603.28750 · HTML: https://mlsanity.com/paper/2603.28750/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 18 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 5 of 5)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels](https://mlsanity.com/paper/2603.19312/) (2026)
- [Unbiased Online Recurrent Optimization](https://mlsanity.com/paper/1702.05043/) (2017)
- [Optimal Kronecker-Sum Approximation of Real Time Recurrent Learning](https://mlsanity.com/paper/1902.03993/) (2019)
- [Joint Online Learning and Decision-making via Dual Mirror Descent](https://mlsanity.com/paper/2104.09750/) (2021)

## Abstract (arXiv)

For three decades online recurrent learning has been assumed to require propagating a Jacobian tensor through the network's dynamics at O(n4) per step. We show it doesn't. Dropping the propagation entirely (d=0, O(n2) memory) matches full RTRL within CI on held-out BCI cross-session drift (TOST equivalent within ± 3 pp at n=20, Adam, float64), and across vanilla-RNN synthetic cells (sine and Lorenz under Adam and SGD) and LSTM/sine under Adam. A decomposition gRTRL = gimm + gpast explains why. On BCI, gpast concentrates in a single direction (top-1 singular fraction 0.62-0.74 across four optimizers, vs 0.333 for gimm), and the four-optimizer full-RTRL-vs-d=0 recovery gap tracks each optimizer's per-layer update-magnitude ratio |Delta Whh|/|Delta Wout| monotonically. A stationary (no-drift) control collapses both concentrations to ~0.6: the drift-specific signal is the differential, not gpast's absolute rank-1 structure. The signature and the behavioral gap both collapse on LSTM, consistent with a mechanism specific to additive linear recurrence. On synthetic sine, gimm is redundant with gpast, which predicts the synthetic null. Full RTRL's one robust advantage is LARS (+17 to +27 pp), but d=0+LARS also fails to adapt independently; the gap is an optimizer×method interaction, not a method-quality claim. We characterize the regime: d=0+Adam+float64 is robust; SGD, Adafactor, and float32 have specific fragilities documented in the paper. On the evaluated cells, the 1000× memory saving at n=1024 (O(n2) vs O(n4)) comes with no measured recovery cost.