mlsanityData to 14 Oct 2026

Immediate Derivatives Suffice for Online Recurrent Adaptation

arXiv 2603.28750 ↗First posted 30 Mar 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

For three decades online recurrent learning has been assumed to require propagating a Jacobian tensor through the network's dynamics at O(n4) per step. We show it doesn't. Dropping the propagation entirely (d=0, O(n2) memory) matches full RTRL within CI on held-out BCI cross-session drift (TOST equivalent within ± 3 pp at n=20, Adam, float64), and across vanilla-RNN synthetic cells (sine and Lorenz under Adam and SGD) and LSTM/sine under Adam. A decomposition gRTRL = gimm + gpast explains why. On BCI, gpast concentrates in a single direction (top-1 singular fraction 0.62-0.74 across four optimizers, vs 0.333 for gimm), and the four-optimizer full-RTRL-vs-d=0 recovery gap tracks each optimizer's per-layer update-magnitude ratio |Delta Whh|/|Delta Wout| monotonically. A stationary (no-drift) control collapses both concentrations to ~0.6: the drift-specific signal is the differential, not gpast's absolute rank-1 structure. The signature and the behavioral gap both collapse on LSTM, consistent with a mechanism specific to additive linear recurrence. On synthetic sine, gimm is redundant with gpast, which predicts the synthetic null. Full RTRL's one robust advantage is LARS (+17 to +27 pp), but d=0+LARS also fails to adapt independently; the gap is an optimizer×method interaction, not a method-quality claim. We characterize the regime: d=0+Adam+float64 is robust; SGD, Adafactor, and float32 have specific fragilities documented in the paper. On the evaluated cells, the 1000× memory saving at n=1024 (O(n2) vs O(n4)) comes with no measured recovery cost.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 5 of its 18 references that are in the map. First posted on arXiv 30 Mar 2026.

Builds on

5 of 18 references.

References · 5

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 273
  3. 2017Unbiased Online Recurrent Optimizationcited by 47
  4. 2019Optimal Kronecker-Sum Approximation of Real Time Recurrent Learningcited by 22
  5. 2021Joint Online Learning and Decision-making via Dual Mirror Descentcited by 8

Built on by

Not cited yet.