# False Memory Formation in Continual Learners Through Imperceptible Backdoor Trigger

arXiv: https://arxiv.org/abs/2202.04479 · HTML: https://mlsanity.com/paper/2202.04479/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 9 Feb 2022.

## Builds on (most cited first, 18 of 18)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [A continual learning survey: Defying forgetting in classification tasks](https://mlsanity.com/paper/1909.08383/) (2019)
- [Generative replay with feedback connections as a general strategy for continual learning](https://mlsanity.com/paper/1809.10635/) (2018)
- [Generative Models from the perspective of Continual Learning](https://mlsanity.com/paper/1812.09111/) (2018)
- [Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks](https://mlsanity.com/paper/2002.07111/) (2020)
- [Generative Feature Replay with Orthogonal Weight Modification for Continual Learning](https://mlsanity.com/paper/2005.03490/) (2020)
- [Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models](https://mlsanity.com/paper/2102.08355/) (2021)

## Built on by (most cited first, 6 of 6)

- [Adversary Aware Continual Learning](https://mlsanity.com/paper/2304.14483/) (2023)
- [Out-of-distribution forgetting: vulnerability of continual learning to intra-class distribution shift](https://mlsanity.com/paper/2306.00427/) (2023)
- [Backdoor Attacks Against Incremental Learners: An Empirical Evaluation Study](https://mlsanity.com/paper/2305.18384/) (2023)
- [PACOL: Poisoning Attacks Against Continual Learners](https://mlsanity.com/paper/2311.10919/) (2023)
- [Attack On Prompt: Backdoor Attack in Prompt-Based Continual Learning](https://mlsanity.com/paper/2406.19753/) (2024)
- [Addressing The Devastating Effects Of Single-Task Data Poisoning In Exemplar-Free Continual Learning](https://mlsanity.com/paper/2507.04106/) (2025)

## Abstract (arXiv)

In this brief, we show that sequentially learning new information presented to a continual (incremental) learning model introduces new security risks: an intelligent adversary can introduce small amount of misinformation to the model during training to cause deliberate forgetting of a specific task or class at test time, thus creating "false memory" about that task. We demonstrate such an adversary's ability to assume control of the model by injecting "backdoor" attack samples to commonly used generative replay and regularization based continual learning approaches using continual learning benchmark variants of MNIST, as well as the more challenging SVHN and CIFAR 10 datasets. Perhaps most damaging, we show this vulnerability to be very acute and exceptionally effective: the backdoor pattern in our attack model can be imperceptible to human eye, can be provided at any point in time, can be added into the training data of even a single possibly unrelated task and can be achieved with as few as just 1% of total training dataset of a single task.