# SYNERgy between SYNaptic consolidation and Experience Replay for general continual learning

arXiv: https://arxiv.org/abs/2206.04016 · HTML: https://mlsanity.com/paper/2206.04016/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 8 Jun 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [Large Scale Incremental Learning](https://mlsanity.com/paper/1905.13260/) (2019)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [Gradient based sample selection for online continual learning](https://mlsanity.com/paper/1903.08671/) (2019)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1805.07810/) (2018)
- [Towards Robust Evaluations of Continual Learning](https://mlsanity.com/paper/1805.09733/) (2018)
- [Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System](https://mlsanity.com/paper/2201.12604/) (2022)
- [Replay in Deep Learning: Current Approaches and Missing Biological Elements](https://mlsanity.com/paper/2104.04132/) (2021)
- [DualNet: Continual Learning, Fast and Slow](https://mlsanity.com/paper/2110.00175/) (2021)
- [Biologically inspired sleep algorithm for artificial neural networks](https://mlsanity.com/paper/1908.02240/) (2019)

## Built on by (most cited first, 10 of 10)

- [BiRT: Bio-inspired Replay in Vision Transformers for Continual Learning](https://mlsanity.com/paper/2305.04769/) (2023)
- [Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective](https://mlsanity.com/paper/2506.03951/) (2025)
- [Continual Learning in the Frequency Domain](https://mlsanity.com/paper/2410.06645/) (2024)
- [TriRE: A Multi-Mechanism Learning Paradigm for Continual Knowledge Retention and Promotion](https://mlsanity.com/paper/2310.08217/) (2023)
- [Dual Cognitive Architecture: Incorporating Biases and Multi-Memory Systems for Lifelong Learning](https://mlsanity.com/paper/2310.11341/) (2023)
- [Beyond Unimodal Learning: The Importance of Integrating Multiple Modalities for Lifelong Learning](https://mlsanity.com/paper/2405.02766/) (2024)
- [IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learning](https://mlsanity.com/paper/2404.18161/) (2024)
- [Continual Vision-and-Language Navigation](https://mlsanity.com/paper/2403.15049/) (2024)
- [Mitigating Interference in the Knowledge Continuum through Attention-Guided Incremental Learning](https://mlsanity.com/paper/2405.13978/) (2024)
- [Continual Learning Beyond Experience Rehearsal and Full Model Surrogates](https://mlsanity.com/paper/2505.21942/) (2025)

## Abstract (arXiv)

Continual learning (CL) in the brain is facilitated by a complex set of mechanisms. This includes the interplay of multiple memory systems for consolidating information as posited by the complementary learning systems (CLS) theory and synaptic consolidation for protecting the acquired knowledge from erasure. Thus, we propose a general CL method that creates a synergy between SYNaptic consolidation and dual memory Experience Replay (SYNERgy). Our method maintains a semantic memory that accumulates and consolidates information across the tasks and interacts with the episodic memory for effective replay. It further employs synaptic consolidation by tracking the importance of parameters during the training trajectory and anchoring them to the consolidated parameters in the semantic memory. To the best of our knowledge, our study is the first to employ dual memory experience replay in conjunction with synaptic consolidation that is suitable for general CL whereby the network does not utilize task boundaries or task labels during training or inference. Our evaluation on various challenging CL scenarios and characteristics analyses demonstrate the efficacy of incorporating both synaptic consolidation and CLS theory in enabling effective CL in DNNs.