# Adversarial Targeted Forgetting in Regularization and Generative Based Continual Learning Models

arXiv: https://arxiv.org/abs/2102.08355 · HTML: https://mlsanity.com/paper/2102.08355/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security.
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 16 Feb 2021.

## Placed in

- LLM safety and security / [Backdoor and data poisoning attacks](https://mlsanity.com/safety/backdoor-and-data-poisoning-attacks/)

## Builds on (most cited first, 17 of 17)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Evasion Attacks against Machine Learning at Test Time](https://mlsanity.com/paper/1708.06131/) (2017)
- [Three scenarios for continual learning](https://mlsanity.com/paper/1904.07734/) (2019)
- [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018)
- [Progress & Compress: A scalable framework for continual learning](https://mlsanity.com/paper/1805.06370/) (2018)
- [Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks](https://mlsanity.com/paper/1804.00792/) (2018)
- [Generative replay with feedback connections as a general strategy for continual learning](https://mlsanity.com/paper/1809.10635/) (2018)
- [Continual Classification Learning Using Generative Models](https://mlsanity.com/paper/1810.10612/) (2018)
- [Targeted Forgetting and False Memory Formation in Continual Learners through Adversarial Backdoor Attacks](https://mlsanity.com/paper/2002.07111/) (2020)
- [Attack Strength vs. Detectability Dilemma in Adversarial Machine Learning](https://mlsanity.com/paper/1802.07295/) (2018)

## Built on by (most cited first, 9 of 9)

- [Persistent Backdoor Attacks in Continual Learning](https://mlsanity.com/paper/2409.13864/) (2024)
- [False Memory Formation in Continual Learners Through Imperceptible Backdoor Trigger](https://mlsanity.com/paper/2202.04479/) (2022)
- [Adversary Aware Continual Learning](https://mlsanity.com/paper/2304.14483/) (2023)
- [Out-of-distribution forgetting: vulnerability of continual learning to intra-class distribution shift](https://mlsanity.com/paper/2306.00427/) (2023)
- [Backdoor Attacks Against Incremental Learners: An Empirical Evaluation Study](https://mlsanity.com/paper/2305.18384/) (2023)
- [Attack On Prompt: Backdoor Attack in Prompt-Based Continual Learning](https://mlsanity.com/paper/2406.19753/) (2024)
- [Noise-Tolerant Coreset-Based Class Incremental Continual Learning](https://mlsanity.com/paper/2504.16763/) (2025)
- [Anchoring Bias: A Persistent Fairness Backdoor Attack against MLLMs under Continual Learning](https://mlsanity.com/paper/2608.21577/) (2026)
- [Robust Dynamic Expansion for Continual Learning under Backdoor Attacks via Purification and Selective Recovery](https://mlsanity.com/paper/2609.06346/) (2026)

## Abstract (arXiv)

Continual (or "incremental") learning approaches are employed when additional knowledge or tasks need to be learned from subsequent batches or from streaming data. However these approaches are typically adversary agnostic, i.e., they do not consider the possibility of a malicious attack. In our prior work, we explored the vulnerabilities of Elastic Weight Consolidation (EWC) to the perceptible misinformation. We now explore the vulnerabilities of other regularization-based as well as generative replay-based continual learning algorithms, and also extend the attack to imperceptible misinformation. We show that an intelligent adversary can take advantage of a continual learning algorithm's capabilities of retaining existing knowledge over time, and force it to learn and retain deliberately introduced misinformation. To demonstrate this vulnerability, we inject backdoor attack samples into the training data. These attack samples constitute the misinformation, allowing the attacker to capture control of the model at test time. We evaluate the extent of this vulnerability on both rotated and split benchmark variants of the MNIST dataset under two important domain and class incremental learning scenarios. We show that the adversary can create a "false memory" about any task by inserting carefully-designed backdoor samples to the test instances of that task thereby controlling the amount of forgetting of any task of its choosing. Perhaps most importantly, we show this vulnerability to be very acute and damaging: the model memory can be easily compromised with the addition of backdoor samples into as little as 1% of the training data, even when the misinformation is imperceptible to human eye.