# Lifelong Learning of Spatiotemporal Representations with Dual-Memory Recurrent Self-Organization

arXiv: https://arxiv.org/abs/1805.10966 · HTML: https://mlsanity.com/paper/1805.10966/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 May 2018.

## Built on by (most cited first, 30 of 38)

- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [CLIP model is an Efficient Continual Learner](https://mlsanity.com/paper/2210.03114/) (2022)
- [Online Continual Learning for Embedded Devices](https://mlsanity.com/paper/2203.10681/) (2022)
- [DualNet: Continual Learning, Fast and Slow](https://mlsanity.com/paper/2110.00175/) (2021)
- [Continual Learning for Recurrent Neural Networks: an Empirical Evaluation](https://mlsanity.com/paper/2103.07492/) (2021)
- [Graph Lifelong Learning: A Survey](https://mlsanity.com/paper/2202.10688/) (2022)
- [Domain-Incremental Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition](https://mlsanity.com/paper/2103.08637/) (2021)
- [Plasticity-Optimized Complementary Networks for Unsupervised Continual Learning](https://mlsanity.com/paper/2309.06086/) (2023)
- [IB-DRR: Incremental Learning with Information-Back Discrete Representation Replay](https://mlsanity.com/paper/2104.10588/) (2021)
- [Reactive Exploration to Cope with Non-Stationarity in Lifelong Reinforcement Learning](https://mlsanity.com/paper/2207.05742/) (2022)
- [Continual Learning in Open-vocabulary Classification with Complementary Memory Systems](https://mlsanity.com/paper/2307.01430/) (2023)
- [Self-Activating Neural Ensembles for Continual Reinforcement Learning](https://mlsanity.com/paper/2301.00141/) (2022)
- [Continual Learning for Autonomous Robots: A Prototype-based Approach](https://mlsanity.com/paper/2404.00418/) (2024)
- [A Continual and Incremental Learning Approach for TinyML On-device Training Using Dataset Distillation and Model Size Adaption](https://mlsanity.com/paper/2409.07114/) (2024)
- [DRILL: Dynamic Representations for Imbalanced Lifelong Learning](https://mlsanity.com/paper/2105.08445/) (2021)
- [A Parameter-free Adaptive Resonance Theory-based Topological Clustering Algorithm Capable of Continual Learning](https://mlsanity.com/paper/2305.01507/) (2023)
- [Towards Fair Affective Robotics: Continual Learning for Mitigating Bias in Facial Expression and Action Unit Recognition](https://mlsanity.com/paper/2103.09233/) (2021)
- [Neural Weight Search for Scalable Task Incremental Learning](https://mlsanity.com/paper/2211.13823/) (2022)
- [Lifelong Machine Learning Potentials](https://mlsanity.com/paper/2303.05911/) (2023)
- [Adiabatic replay for continual learning](https://mlsanity.com/paper/2303.13157/) (2023)
- [Continual Learning from Synthetic Data for a Humanoid Exercise Robot](https://mlsanity.com/paper/2102.10034/) (2021)
- [Towards Heterogeneous Clients with Elastic Federated Learning](https://mlsanity.com/paper/2106.09433/) (2021)
- [Lifelong 3D Object Recognition and Grasp Synthesis Using Dual Memory Recurrent Self-Organization Networks](https://mlsanity.com/paper/2109.11544/) (2021)
- [One-shot learning for the long term: consolidation with an artificial hippocampal algorithm](https://mlsanity.com/paper/2102.07503/) (2021)
- [Target Layer Regularization for Continual Learning Using Cramer-Wold Generator](https://mlsanity.com/paper/2111.07928/) (2021)
- [Object Pursuit: Building a Space of Objects via Discriminative Weight Generation](https://mlsanity.com/paper/2112.07954/) (2021)
- [Robust Continual Learning through a Comprehensively Progressive Bayesian Neural Network](https://mlsanity.com/paper/2202.13369/) (2022)
- [Learning Citywide Patterns of Life from Trajectory Monitoring](https://mlsanity.com/paper/2206.15352/) (2022)
- [Architect, Regularize and Replay (ARR): a Flexible Hybrid Approach for Continual Learning](https://mlsanity.com/paper/2301.02464/) (2023)
- [Explainable Lifelong Stream Learning Based on "Glocal" Pairwise Fusion](https://mlsanity.com/paper/2306.13410/) (2023)

## Abstract (arXiv)

Artificial autonomous agents and robots interacting in complex environments are required to continually acquire and fine-tune knowledge over sustained periods of time. The ability to learn from continuous streams of information is referred to as lifelong learning and represents a long-standing challenge for neural network models due to catastrophic forgetting. Computational models of lifelong learning typically alleviate catastrophic forgetting in experimental scenarios with given datasets of static images and limited complexity, thereby differing significantly from the conditions artificial agents are exposed to. In more natural settings, sequential information may become progressively available over time and access to previous experience may be restricted. In this paper, we propose a dual-memory self-organizing architecture for lifelong learning scenarios. The architecture comprises two growing recurrent networks with the complementary tasks of learning object instances (episodic memory) and categories (semantic memory). Both growing networks can expand in response to novel sensory experience: the episodic memory learns fine-grained spatiotemporal representations of object instances in an unsupervised fashion while the semantic memory uses task-relevant signals to regulate structural plasticity levels and develop more compact representations from episodic experience. For the consolidation of knowledge in the absence of external sensory input, the episodic memory periodically replays trajectories of neural reactivations. We evaluate the proposed model on the CORe50 benchmark dataset for continuous object recognition, showing that we significantly outperform current methods of lifelong learning in three different incremental learning scenarios