# Learning Long-Range Dependencies with Temporal Predictive Coding

arXiv: https://arxiv.org/abs/2602.18131 · HTML: https://mlsanity.com/paper/2602.18131/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 66 references that are in the map.
- First posted on arXiv 20 Feb 2026.

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks](https://mlsanity.com/paper/1506.03099/) (2015)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagation](https://mlsanity.com/paper/1602.05179/) (2016)
- [CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB](https://mlsanity.com/paper/1911.04944/) (2019)
- [Efficient Neuromorphic Signal Processing with Loihi 2](https://mlsanity.com/paper/2111.03746/) (2021)
- [Predictive Coding Approximates Backprop along Arbitrary Computation Graphs](https://mlsanity.com/paper/2006.04182/) (2020)
- [Predictive Coding: a Theoretical and Experimental Review](https://mlsanity.com/paper/2107.12979/) (2021)
- [Unbiased Online Recurrent Optimization](https://mlsanity.com/paper/1702.05043/) (2017)
- [A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks](https://mlsanity.com/paper/1907.02649/) (2019)
- [Learning on Arbitrary Graph Topologies via Predictive Coding](https://mlsanity.com/paper/2201.13180/) (2022)
- [Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations](https://mlsanity.com/paper/1810.07411/) (2018)
- [Benchmarking Predictive Coding Networks -- Made Simple](https://mlsanity.com/paper/2407.01163/) (2024)
- [Approximating Real-Time Recurrent Learning with Random Kronecker Factors](https://mlsanity.com/paper/1805.10842/) (2018)
- [Online learning of long-range dependencies](https://mlsanity.com/paper/2305.15947/) (2023)
- [SpiNNaker2: A Large-Scale Neuromorphic System for Event-Based and Asynchronous Machine Learning](https://mlsanity.com/paper/2401.04491/) (2024)
- [Reverse Differentiation via Predictive Coding](https://mlsanity.com/paper/2103.04689/) (2021)
- [A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks](https://mlsanity.com/paper/2212.00720/) (2022)
- [Neuromorphic Hardware In The Loop: Training a Deep Spiking Network on the BrainScaleS Wafer-Scale System](https://mlsanity.com/paper/1703.01909/) (2017)
- [Exploring the Promise and Limits of Real-Time Recurrent Learning](https://mlsanity.com/paper/2305.19044/) (2023)
- [Predictive Coding beyond Gaussian Distributions](https://mlsanity.com/paper/2211.03481/) (2022)
- [Sequential Memory with Temporal Predictive Coding](https://mlsanity.com/paper/2305.11982/) (2023)
- [Demonstrating Advantages of Neuromorphic Computation: A Pilot Study](https://mlsanity.com/paper/1811.03618/) (2018)

## Abstract (arXiv)

Temporal Predictive Coding provides a layer-local, parallelisable mechanism for learning in recurrent systems, making it an attractive candidate for online local learning on neuromorphic and edge hardware. However, its recurrent parameter update captures only local temporal relationships, neglecting the historic influence of parameters along the latent-state trajectory, and therefore struggles to assign credit over longer temporal horizons. This work combines for the first time Temporal Predictive Coding with Real-Time Recurrent Learning (tPC-RTRL), incorporating an online influence matrix that tracks this historic effect whilst preserving the spatial and temporal locality properties valued by neuromorphic implementations. Under explicit assumptions, we prove that tPC-RTRL recovers the gradients of backpropagation-through-time exactly. Empirically, a near-equivalence holds across several tasks of varying scale and complexity, including byte-level language modelling on WikiText-103 (tPC-RTRL vs. BPTT: 1.865 vs. 1.864 validation BPC), English--French translation on a CCMatrix subset (20.23 vs. 20.29 BLEU), and a realistic nanodrone system-identification benchmark (0.506m vs. 0.505m mean position error). Finally, we show that the iterative inference mechanism used during training can be reused at deployment time to incorporate intermittent state observations, halving final-position error relative to open-loop rollout on the nanodrone task (0.402m vs. 0.805m) and suggesting a path towards unifying learning and filtering within the same computational framework.