# Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks

arXiv: https://arxiv.org/abs/2002.07111 · HTML: https://mlsanity.com/paper/2002.07111/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Feb 2020.

## Built on by (most cited first, 15 of 15)

- [Persistent Backdoor Attacks in Continual Learning](https://mlsanity.com/paper/2409.13864/) (2024)
- [Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models](https://mlsanity.com/paper/2102.08355/) (2021)
- [BrainWash: A Poisoning Attack to Forget in Continual Learning](https://mlsanity.com/paper/2311.11995/) (2023)
- [False Memory Formation in Continual Learners Through Imperceptible Backdoor Trigger](https://mlsanity.com/paper/2202.04479/) (2022)
- [Adversarial Robust Memory-Based Continual Learner](https://mlsanity.com/paper/2311.17608/) (2023)
- [Adversary Aware Continual Learning](https://mlsanity.com/paper/2304.14483/) (2023)
- [Out-of-distribution forgetting: vulnerability of continual learning to intra-class distribution shift](https://mlsanity.com/paper/2306.00427/) (2023)
- [Improving Data-aware and Parameter-aware Robustness for Continual Learning](https://mlsanity.com/paper/2405.17054/) (2024)
- [Backdoor Attacks Against Incremental Learners: An Empirical Evaluation Study](https://mlsanity.com/paper/2305.18384/) (2023)
- [PACOL: Poisoning Attacks Against Continual Learners](https://mlsanity.com/paper/2311.10919/) (2023)
- [Attack On Prompt: Backdoor Attack in Prompt-Based Continual Learning](https://mlsanity.com/paper/2406.19753/) (2024)
- [Noise-Tolerant Coreset-Based Class Incremental Continual Learning](https://mlsanity.com/paper/2504.16763/) (2025)
- [Addressing The Devastating Effects Of Single-Task Data Poisoning In Exemplar-Free Continual Learning](https://mlsanity.com/paper/2507.04106/) (2025)
- [Theory of Continual Learning Against Data Poisoning Attacks](https://mlsanity.com/paper/2606.29841/) (2026)
- [Adaptive Intrusion Detection System using Transformer-Based Neural Networks and Continual Learning Approach with Adversarial Investigation](https://mlsanity.com/paper/2608.04602/) (2026)

## Abstract (arXiv)

Artificial neural networks are well-known to be susceptible to catastrophic forgetting when continually learning from sequences of tasks. Various continual (or "incremental") learning approaches have been proposed to avoid catastrophic forgetting, but they are typically adversary agnostic, i.e., they do not consider the possibility of a malicious attack. In this effort, we explore the vulnerability of Elastic Weight Consolidation (EWC), a popular continual learning algorithm for avoiding catastrophic forgetting. We show that an intelligent adversary can bypass the EWC's defenses, and instead cause gradual and deliberate forgetting by introducing small amounts of misinformation to the model during training. We demonstrate such an adversary's ability to assume control of the model via injection of "backdoor" attack samples on both permuted and split benchmark variants of the MNIST dataset. Importantly, once the model has learned the adversarial misinformation, the adversary can then control the amount of forgetting of any task. Equivalently, the malicious actor can create a "false memory" about any task by inserting carefully-designed backdoor samples to any fraction of the test instances of that task. Perhaps most damaging, we show this vulnerability to be very acute; neural network memory can be easily compromised with the addition of backdoor samples into as little as 1% of the training data of even a single task.