mlsanityData to 7 Oct 2026

Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models

arXiv 2103.15543 ↗First posted 29 Mar 2021cs.CLcited by 96

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Recent studies have revealed a security threat to natural language processing (NLP) models, called the Backdoor Attack. Victim models can maintain competitive performance on clean samples while behaving abnormally on samples with a specific trigger word inserted. Previous backdoor attacking methods usually assume that attackers have a certain degree of data knowledge, either the dataset which users would use or proxy datasets for a similar task, for implementing the data poisoning procedure. However, in this paper, we find that it is possible to hack the model in a data-free way by modifying one single word embedding vector, with almost no accuracy sacrificed on clean samples. Experimental results on sentiment analysis and sentence-pair classification tasks show that our method is more efficient and stealthier. We hope this work can raise the awareness of such a critical security risk hidden in the embedding layers of NLP models. Our code is available at https://github.com/lancopku/Embedding-Poisoning.

Key facts. Cited by 96 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 23 of its 38 references that are in the map. First posted on arXiv 29 Mar 2021.

Builds on

23 of 38 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  11. 2019Hidden Trigger Backdoor Attackscited by 359
  12. 2020Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networkscited by 341
11 more
  1. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  2. 2020Input-Aware Dynamic Backdoor Attackcited by 304
  3. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  4. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 119
  5. 2020Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Casescited by 93
  6. 2020Trojaning Language Models for Fun and Profitcited by 80
  7. 2020Rethinking the Trigger of Backdoor Attackcited by 76
  8. 2020Can Adversarial Weight Perturbations Inject Neural Backdoors?cited by 51
  9. 2019NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanationscited by 49
  10. 2020Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencodercited by 27
  11. 2020Humpty Dumpty: Controlling Word Meanings via Corpus Poisoningcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 17; 2023: 23; 2024: 32; 2025: 12; 2026: 4.

96 papers.

Sort:

By area:

  1. 2023Poisoning Language Models During Instruction Tuningcited by 254
  2. 2023Large Language Model Alignment: A Surveycited by 241
  3. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  4. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  5. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  6. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  7. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 97
  8. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  9. 2021BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Modelscited by 76
  10. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarkscited by 70
  11. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  12. 2024Instructional Fingerprinting of Large Language Modelscited by 64
38 more
  1. 2023Composite Backdoor Attacks Against Large Language Modelscited by 63
  2. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labelscited by 61
  3. 2023Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermarkcited by 61
  4. 2022BadPrompt: Backdoor Attacks on Continuous Promptscited by 59
  5. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Modelscited by 50
  6. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  7. 2022A Study of the Attention Abnormality in Trojaned BERTscited by 47
  8. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injectioncited by 47
  9. 2024Persistent Pre-Training Poisoning of LLMscited by 46
  10. 2022Constrained Optimization with Dynamic Bound-scaling for Effective NLPBackdoor Defensecited by 44
  11. 2022Watermarking Pre-trained Language Models with Backdooringcited by 44
  12. 2023UNICORN: A Unified Backdoor Trigger Inversion Frameworkcited by 44
  13. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  14. 2022Rethinking the Reverse-engineering of Trojan Triggerscited by 37
  15. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  16. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  17. 2023Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attackscited by 32
  18. 2023Attention-Enhancing Backdoor Attacks Against BERT-based Modelscited by 32
  19. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  20. 2021Defending Against Backdoor Attacks in Natural Language Generationcited by 30
  21. 2023ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Triggercited by 29
  22. 2023Training-free Lexical Backdoor Attacks on Language Modelscited by 27
  23. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 27
  24. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 26
  25. 2023Defending against Insertion-based Textual Backdoor Attacks via Attributioncited by 24
  26. 2024Task-Agnostic Detector for Insertion-Based Backdoor Attackscited by 23
  27. 2025DemonAgent: Dynamically Encrypted Multi-Backdoor Implantation Attack on LLM-based Agentcited by 23
  28. 2021How to Inject Backdoors with Better Consistency: Logit Anchoring on Clean Datacited by 22
  29. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 19
  30. 2022Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attackscited by 18
  31. 2024Double-I Watermark: Protecting Model Copyright for LLM Fine-tuningcited by 18
  32. 2023ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLPcited by 16
  33. 2023TextGuard: Provable Defense against Backdoor Attacks on Text Classificationcited by 14
  34. 2024Decoding the AI Pen: Techniques and Challenges in Detecting AI-Generated Textcited by 14
  35. 2022A Survey on Backdoor Attack and Defense in Natural Language Processingcited by 13
  36. 2024Cognitive Overload Attack:Prompt Injection for Long Contextcited by 13
  37. 2023Watermarking Vision-Language Pre-trained Models for Multi-modal Embedding as a Servicecited by 11
  38. 2024Exploring Backdoor Vulnerabilities of Chat Modelscited by 11

50 most cited of 96 All 96 (JSON)