# Continual Learning of Recurrent Neural Networks by Locally Aligning Distributed Representations

arXiv: https://arxiv.org/abs/1810.07411 · HTML: https://mlsanity.com/paper/1810.07411/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2018.

## Built on by (most cited first, 30 of 32)

- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Continual Learning for Recurrent Neural Networks: an Empirical Evaluation](https://mlsanity.com/paper/2103.07492/) (2021)
- [Brain-inspired Computational Intelligence via Predictive Coding](https://mlsanity.com/paper/2308.07870/) (2023)
- [A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks](https://mlsanity.com/paper/2212.00720/) (2022)
- [Class Incremental Learning with Self-Supervised Pre-Training and Prototype Learning](https://mlsanity.com/paper/2308.02346/) (2023)
- [Sequential Memory with Temporal Predictive Coding](https://mlsanity.com/paper/2305.11982/) (2023)
- [How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective](https://mlsanity.com/paper/2106.08453/) (2021)
- [Backprop-Free Reinforcement Learning with Active Neural Generative Coding](https://mlsanity.com/paper/2107.07046/) (2021)
- [Online Spatio-Temporal Learning with Target Projection](https://mlsanity.com/paper/2304.05124/) (2023)
- [Continual Competitive Memory: A Neural System for Online Task-Free Lifelong Learning](https://mlsanity.com/paper/2106.13300/) (2021)
- [Continual Learning of Dynamical Systems with Competitive Federated Reservoir Computing](https://mlsanity.com/paper/2206.13336/) (2022)
- [Active Predicting Coding: Brain-Inspired Reinforcement Learning for Sparse Reward Robotic Control Problems](https://mlsanity.com/paper/2209.09174/) (2022)
- [Fixed-Weight Difference Target Propagation](https://mlsanity.com/paper/2212.10352/) (2022)
- [Training Deep Architectures Without End-to-End Backpropagation: A Survey on the Provably Optimal Methods](https://mlsanity.com/paper/2101.03419/) (2021)
- [Robust Graph Representation Learning via Predictive Coding](https://mlsanity.com/paper/2212.04656/) (2022)
- [Active Sensing with Predictive Coding and Uncertainty Minimization](https://mlsanity.com/paper/2307.00668/) (2023)
- [Maze Learning using a Hyperdimensional Predictive Processing Cognitive Architecture](https://mlsanity.com/paper/2204.00619/) (2022)
- [AdaptCL: Adaptive Continual Learning for Tackling Heterogeneity in Sequential Datasets](https://mlsanity.com/paper/2207.11005/) (2022)
- [Tight Stability, Convergence, and Robustness Bounds for Predictive Coding Networks](https://mlsanity.com/paper/2410.04708/) (2024)
- [Bidirectional Interaction between Visual and Motor Generative Models using Predictive Coding and Active Inference](https://mlsanity.com/paper/2104.09163/) (2021)
- [Towards a Predictive Processing Implementation of the Common Model of Cognition](https://mlsanity.com/paper/2105.07308/) (2021)
- [A Neuro-mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization](https://mlsanity.com/paper/2310.15177/) (2023)
- [A Predictive Coding Account for Chaotic Itinerancy](https://mlsanity.com/paper/2106.08937/) (2021)
- [Biologically-Inspired Continual Learning of Human Motion Sequences](https://mlsanity.com/paper/2211.05231/) (2022)
- [Quantized Approximately Orthogonal Recurrent Neural Networks](https://mlsanity.com/paper/2402.04012/) (2024)
- [Neuro-mimetic Task-free Unsupervised Online Learning with Continual Self-Organizing Maps](https://mlsanity.com/paper/2402.12465/) (2024)
- [Bridging Predictive Coding and MDL: A Two-Part Code Framework for Deep Learning](https://mlsanity.com/paper/2505.14635/) (2025)
- [Like a bilingual baby: The advantage of visually grounding a bilingual language model](https://mlsanity.com/paper/2210.05487/) (2022)
- [Predictive Attractor Models](https://mlsanity.com/paper/2410.02430/) (2024)
- [Learning Long-Range Dependencies with Temporal Predictive Coding](https://mlsanity.com/paper/2602.18131/) (2026)

## Abstract (arXiv)

Temporal models based on recurrent neural networks have proven to be quite powerful in a wide variety of applications. However, training these models often relies on back-propagation through time, which entails unfolding the network over many time steps, making the process of conducting credit assignment considerably more challenging. Furthermore, the nature of back-propagation itself does not permit the use of non-differentiable activation functions and is inherently sequential, making parallelization of the underlying training process difficult. Here, we propose the Parallel Temporal Neural Coding Network (P-TNCN), a biologically inspired model trained by the learning algorithm we call Local Representation Alignment. It aims to resolve the difficulties and problems that plague recurrent networks trained by back-propagation through time. The architecture requires neither unrolling in time nor the derivatives of its internal activation functions. We compare our model and learning procedure to other back-propagation through time alternatives (which also tend to be computationally expensive), including real-time recurrent learning, echo state networks, and unbiased online recurrent optimization. We show that it outperforms these on sequence modeling benchmarks such as Bouncing MNIST, a new benchmark we denote as Bouncing NotMNIST, and Penn Treebank. Notably, our approach can in some instances outperform full back-propagation through time as well as variants such as sparse attentive back-tracking. Significantly, the hidden unit correction phase of P-TNCN allows it to adapt to new datasets even if its synaptic weights are held fixed (zero-shot adaptation) and facilitates retention of prior generative knowledge when faced with a task sequence. We present results that show the P-TNCN's ability to conduct zero-shot adaptation and online continual sequence modeling.