mlsanityData to 7 Oct 2026

Learning Long-Range Dependencies with Temporal Predictive Coding

arXiv 2602.18131 ↗First posted 20 Feb 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Temporal Predictive Coding provides a layer-local, parallelisable mechanism for learning in recurrent systems, making it an attractive candidate for online local learning on neuromorphic and edge hardware. However, its recurrent parameter update captures only local temporal relationships, neglecting the historic influence of parameters along the latent-state trajectory, and therefore struggles to assign credit over longer temporal horizons. This work combines for the first time Temporal Predictive Coding with Real-Time Recurrent Learning (tPC-RTRL), incorporating an online influence matrix that tracks this historic effect whilst preserving the spatial and temporal locality properties valued by neuromorphic implementations. Under explicit assumptions, we prove that tPC-RTRL recovers the gradients of backpropagation-through-time exactly. Empirically, a near-equivalence holds across several tasks of varying scale and complexity, including byte-level language modelling on WikiText-103 (tPC-RTRL vs. BPTT: 1.865 vs. 1.864 validation BPC), English--French translation on a CCMatrix subset (20.23 vs. 20.29 BLEU), and a realistic nanodrone system-identification benchmark (0.506m vs. 0.505m mean position error). Finally, we show that the iterative inference mechanism used during training can be reused at deployment time to incorporate intermittent state observations, halving final-position error relative to open-loop rollout on the nanodrone task (0.402m vs. 0.805m) and suggesting a path towards unifying learning and filtering within the same computational framework.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 33 of its 66 references that are in the map. First posted on arXiv 20 Feb 2026.

Builds on

33 of 66 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  8. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  9. 2019Green AIcited by 620
  10. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  11. 2016Equilibrium Propagation: Bridging the Gap Between Energy-Based Models and Backpropagationcited by 251
  12. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
21 more
  1. 2021Efficient Neuromorphic Signal Processing with Loihi 2cited by 90
  2. 2020Predictive Coding Approximates Backprop along Arbitrary Computation Graphscited by 89
  3. 2021Predictive Coding: a Theoretical and Experimental Reviewcited by 74
  4. 2017Unbiased Online Recurrent Optimizationcited by 46
  5. 2019A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networkscited by 37
  6. 2022Learning on Arbitrary Graph Topologies via Predictive Codingcited by 33
  7. 2018Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representationscited by 32
  8. 2018Approximating Real-Time Recurrent Learning with Random Kronecker Factorscited by 29
  9. 2024Benchmarking Predictive Coding Networks -- Made Simplecited by 29
  10. 2023Online learning of long-range dependenciescited by 25
  11. 2024SpiNNaker2: A Large-Scale Neuromorphic System for Event-Based and Asynchronous Machine Learningcited by 24
  12. 2021Reverse Differentiation via Predictive Codingcited by 23
  13. 2022A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networkscited by 21
  14. 2017Neuromorphic Hardware In The Loop: Training a Deep Spiking Network on the BrainScaleS Wafer-Scale Systemcited by 19
  15. 2023Exploring the Promise and Limits of Real-Time Recurrent Learningcited by 19
  16. 2022Predictive Coding beyond Gaussian Distributionscited by 16
  17. 2018Demonstrating Advantages of Neuromorphic Computation: A Pilot Studycited by 14
  18. 2023Sequential Memory with Temporal Predictive Codingcited by 14
  19. 2023Understanding and Improving Optimization in Predictive Coding Networkscited by 13
  20. 2024Only Strict Saddles in the Energy Landscape of Predictive Coding Networks?cited by 10
  21. 2024Tight Stability, Convergence, and Robustness Bounds for Predictive Coding Networkscited by 3

Built on by

Not cited yet.