mlsanityData to 14 Oct 2026

Data-centric NLP Backdoor Defense from the Lens of Memorization

arXiv 2409.14200 ↗First posted 21 Sep 2024cs.CL cs.CR cs.LGcited by 6

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Backdoor attack is a severe threat to the trustworthiness of DNN-based language models. In this paper, we first extend the definition of memorization of language models from sample-wise to more fine-grained sentence element-wise (e.g., word, phrase, structure, and style), and then point out that language model backdoors are a type of element-wise memorization. Through further analysis, we find that the strength of such memorization is positively correlated to the frequency of duplicated elements in the training dataset. In conclusion, duplicated sentence elements are necessary for successful backdoor attacks. Based on this, we propose a data-centric defense. We first detect trigger candidates in training data by finding memorizable elements, i.e., duplicated elements, and then confirm real triggers by testing if the candidates can activate backdoor behaviors (i.e., malicious elements). Results show that our method outperforms state-of-the-art defenses in defending against different types of NLP backdoors.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 55 of its 78 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

55 of 78 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection31 Jul
  2. 2023NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models28 May
  3. 2023Backdoor Attacks with Input-unique Triggers in NLP25 Mar
  4. 2023Robust Contrastive Language-Image Pre-training against Data Poisoning and Backdoor Attacks13 Mar
  5. 2022Rethinking the Reverse-engineering of Trojan Triggers27 Oct
  6. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks17 Jun
  7. 2022WeDef: Weakly Supervised Backdoor Defense for Text Classification24 May
  8. 2022Constrained Optimization with Dynamic Bound-scaling for Effective NLPBackdoor Defense11 Feb
  9. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models15 Oct
  10. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer14 Oct
  11. 2021BFClass: A Backdoor-free Text Classification Framework22 Sep
  12. 2021Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution11 Jun
4 more
  1. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May
  2. 2021Hidden Backdoors in Human-Centric Language Models1 May
  3. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models29 Mar
  4. 2021T-Miner: A Generative Approach to Defend Against Trojan Attacks on DNN-based Text Classification7 Mar

Other references · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  8. 2020Extracting Training Data from Large Language Modelscited by 2,189
  9. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
  10. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,055
  11. 2017A Closer Look at Memorization in Deep Networkscited by 942
  12. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
27 more
  1. 2017Automated Hate Speech Detection and the Problem of Offensive Languagecited by 736
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  3. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  4. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
  5. 2018Spectral Signatures in Backdoor Attackscited by 484
  6. 2019STRIP: A Defence Against Trojan Attacks on Deep Neural Networkscited by 459
  7. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 341
  8. 2019Label-Consistent Backdoor Attackscited by 296
  9. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
  10. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 263
  11. 2021Anti-Backdoor Learning: Training Clean Models on Poisoned Datacited by 263
  12. 2019A backdoor attack against LSTM-based text classification systemscited by 227
  13. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 211
  14. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 183
  15. 2017Neural Trojanscited by 153
  16. 2022Backdoor Defense via Decoupling the Training Processcited by 153
  17. 2021SPECTRE: Defending Against Backdoor Attacks Using Robust Statisticscited by 130
  18. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  19. 2020Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identificationcited by 86
  20. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 86
  21. 2022BppAttack: Stealthy and Efficient Trojan Attacks against Deep Neural Networks via Image Quantization and Contrastive Adversarial Learningcited by 86
  22. 2019Design and Evaluation of a Multi-Domain Trojan Detection Method on Deep Neural Networkscited by 71
  23. 2020Fast and Accurate Neural CRF Constituency Parsingcited by 50
  24. 2021Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attackscited by 49
  25. 2023UNICORN: A Unified Backdoor Trigger Inversion Frameworkcited by 46
  26. 2022Training with More Confidence: Mitigating Injected and Natural Backdoors During Trainingcited by 38
  27. 2021Excess Capacity and Backdoor Poisoningcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 2.

6 papers.

  1. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 369
  2. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  3. 2024Robot Collapse: Supply Chain Backdoor Attacks Against VLM-based Robotic Manipulationcited by 15
  4. 2025Backdoor Samples Detection Based on Perturbation Discrepancy Consistency in Pre-trained Language Modelscited by 0
  5. 2026TraceGuard: Adaptive Multimodal Poison Filtering through Cross-Feature Rank Agreementcited by 0
  6. 2026Localize-and-Detect: Auditing Task-Level Poisoning in Instruction-Tuned Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Models
  2. 2022A Survey on Backdoor Attack and Defense in Natural Language Processing
  3. 2024Neutralizing Backdoors through Information Conflicts for Large Language Models
  4. 2024SEEP: Training Dynamics Grounds Latent Representation Search for Mitigating Backdoor Poisoning Attacks
  5. 2025Backdoor Token Unlearning: Exposing and Defending Backdoors in Pretrained Language Models
  6. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injection
  7. 2021BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Models
  8. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Models