# Sparse Coding in a Dual Memory System for Lifelong Learning

arXiv: https://arxiv.org/abs/2301.05058 · HTML: https://mlsanity.com/paper/2301.05058/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 28 Dec 2022.

## Builds on (most cited first, 27 of 27)

- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Online Continual Learning with Maximally Interfered Retrieval](https://mlsanity.com/paper/1908.04742/) (2019)
- [Orthogonal Gradient Descent for Continual Learning](https://mlsanity.com/paper/1910.07104/) (2019)
- [Selective Experience Replay for Lifelong Learning](https://mlsanity.com/paper/1802.10269/) (2018)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Understanding the Role of Training Regimes in Continual Learning](https://mlsanity.com/paper/2006.06958/) (2020)
- [Towards Robust Evaluations of Continual Learning](https://mlsanity.com/paper/1805.09733/) (2018)
- [Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System](https://mlsanity.com/paper/2201.12604/) (2022)
- [Encoder Based Lifelong Learning](https://mlsanity.com/paper/1704.01920/) (2017)
- [Enhancing Adversarial Defense by k-Winners-Take-All](https://mlsanity.com/paper/1905.10510/) (2019)
- [How Can We Be So Dense? The Benefits of Using Highly Sparse Representations](https://mlsanity.com/paper/1903.11257/) (2019)
- [Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting](https://mlsanity.com/paper/2010.01528/) (2020)
- [Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments](https://mlsanity.com/paper/2201.00042/) (2021)
- [Sparsity and Heterogeneous Dropout for Continual Learning in the Null Space of Neural Activations](https://mlsanity.com/paper/2203.06514/) (2022)
- [Consistency is the key to further mitigating catastrophic forgetting in continual learning](https://mlsanity.com/paper/2207.04998/) (2022)
- [Knowledge Distillation Beyond Model Compression](https://mlsanity.com/paper/2007.01922/) (2020)
- [Biologically inspired sleep algorithm for artificial neural networks](https://mlsanity.com/paper/1908.02240/) (2019)

## Built on by (most cited first, 16 of 16)

- [Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy](https://mlsanity.com/paper/2312.10549/) (2023)
- [Handling Out-of-Distribution Data: A Survey](https://mlsanity.com/paper/2507.21160/) (2025)
- [Continual Learning in the Frequency Domain](https://mlsanity.com/paper/2410.06645/) (2024)
- [Elephant Neural Networks: Born to Be a Continual Learner](https://mlsanity.com/paper/2310.01365/) (2023)
- [EVCL: Elastic Variational Continual Learning with Weight Consolidation](https://mlsanity.com/paper/2406.15972/) (2024)
- [Continual Learning in Open-vocabulary Classification with Complementary Memory Systems](https://mlsanity.com/paper/2307.01430/) (2023)
- [Continual Learning with Strong Experience Replay](https://mlsanity.com/paper/2305.13622/) (2023)
- [Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration](https://mlsanity.com/paper/2403.11373/) (2024)
- [Low-redundancy Distillation for Continual Learning](https://mlsanity.com/paper/2309.16117/) (2023)
- [MemoCue: Empowering LLM-Based Agents for Human Memory Recall via Strategy-Guided Querying](https://mlsanity.com/paper/2507.23633/) (2025)
- [Continual Learning of Domain-Invariant Representations](https://mlsanity.com/paper/2605.15775/) (2026)
- [Prior-Free Continual Learning with Unlabeled Data in the Wild](https://mlsanity.com/paper/2310.10417/) (2023)
- [Continual Learning Beyond Experience Rehearsal and Full Model Surrogates](https://mlsanity.com/paper/2505.21942/) (2025)
- [Efficient Reinforcement Learning by Reducing Forgetting with Elephant Activation Functions](https://mlsanity.com/paper/2509.19159/) (2025)
- [Dynamical Adapter Fusion: Constructing A Global Adapter for Pre-Trained Model-based Class-Incremental Learning](https://mlsanity.com/paper/2601.21341/) (2026)
- [Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning](https://mlsanity.com/paper/2601.21345/) (2026)

## Abstract (arXiv)

Efficient continual learning in humans is enabled by a rich set of neurophysiological mechanisms and interactions between multiple memory systems. The brain efficiently encodes information in non-overlapping sparse codes, which facilitates the learning of new associations faster with controlled interference with previous associations. To mimic sparse coding in DNNs, we enforce activation sparsity along with a dropout mechanism which encourages the model to activate similar units for semantically similar inputs and have less overlap with activation patterns of semantically dissimilar inputs. This provides us with an efficient mechanism for balancing the reusability and interference of features, depending on the similarity of classes across tasks. Furthermore, we employ sparse coding in a multiple-memory replay mechanism. Our method maintains an additional long-term semantic memory that aggregates and consolidates information encoded in the synaptic weights of the working model. Our extensive evaluation and characteristics analysis show that equipped with these biologically inspired mechanisms, the model can further mitigate forgetting.