# Prospective Coding Improves Learning in Deep Continuous-Time Recurrent Networks

arXiv: https://arxiv.org/abs/2609.04134 · HTML: https://mlsanity.com/paper/2609.04134/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 3 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [Mamba-3: Improved Sequence Modeling using State Space Principles](https://mlsanity.com/paper/2603.15569/) (2026)
- [Self-learning Machines based on Hamiltonian Echo Backpropagation](https://mlsanity.com/paper/2103.04992/) (2021)
- [Online learning of long-range dependencies](https://mlsanity.com/paper/2305.15947/) (2023)
- [Predictive information in a sensory population](https://mlsanity.com/paper/1307.0225/) (2013)
- [Latent Equilibrium: A unified learning theory for arbitrarily fast computation with arbitrarily slow neurons](https://mlsanity.com/paper/2110.14549/) (2021)
- [Backpropagation through space, time, and the brain](https://mlsanity.com/paper/2403.16933/) (2024)
- [Learning long range dependencies through time reversal symmetry breaking](https://mlsanity.com/paper/2506.05259/) (2025)
- [Equilibrium Propagation for Learning in Lagrangian Dynamical Systems](https://mlsanity.com/paper/2505.07363/) (2025)
- [Unconditional stability of a recurrent neural circuit implementing divisive normalization](https://mlsanity.com/paper/2409.18946/) (2024)
- [Prospective and retrospective coding in cortical neurons](https://mlsanity.com/paper/2405.14810/) (2024)
- [Teaching signal synchronization in deep neural networks with prospective neurons](https://mlsanity.com/paper/2511.14917/) (2025)

## Abstract (arXiv)

Temporal integration gives continuous-time recurrent networks memory, but in deep stacks it also delays bottom-up signals and attenuates top-down errors. We develop Recursive Quadrature Filters (RQFs), biologically motivated complex-valued temporal filters that are a special case of diagonal state-space models (SSMs), and ask whether this failure mode can be addressed by making each layer's bottom-up input prospective. Starting from an energy model, we derive the RQF dynamics and show that each RQF is a band-pass filter whose learnable parameters control its tuning frequency and bandwidth. We then make each layer's bottom-up input prospective using a parameter-free two-tap update that leaves the recurrent transition and parallel scan unchanged. We extend this correction to general diagonal SSMs and show that it mitigates depth-dependent gradient attenuation when temporal gradients are truncated, i.e., spatial-only backpropagation. We evaluate the intervention in RQFs, S5, and ORGaNICs (a nonlinear gated RNN) trained using full backpropagation through time (BPTT) and spatial-only backpropagation. Under full BPTT, prospective variants match or outperform their non-prospective controls in every model and configuration. A non-residual width-32 six-layer RQF reaches 96.09% accuracy on raw-audio Speech Commands with 31.9k parameters; a width-64 six-layer RQF reaches 83.56% on the 16,384-step Path-X task. These results identify RQFs as a parameter-efficient recurrent substrate and prospective-input coding as an input-side correction for deep continuous-time recurrent networks.