mlsanityData to 7 Oct 2026

Weight Poisoning Attacks on Pre-trained Models

arXiv 2004.06660 ↗First posted 14 Apr 2020cs.LG cs.CL cs.CRcited by 318

Not on the map. Why?

Cited by 161 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, NLP has seen a surge in the usage of large pre-trained models. Users download weights of models pre-trained on large datasets, then fine-tune the weights on a task of their choice. This raises the question of whether downloading untrusted pre-trained weights can pose a security threat. In this paper, we show that it is possible to construct ``weight poisoning'' attacks where pre-trained weights are injected with vulnerabilities that expose ``backdoors'' after fine-tuning, enabling the attacker to manipulate the model prediction simply by injecting an arbitrary keyword. We show that by applying a regularization method, which we call RIPPLe, and an initialization procedure, which we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat. Finally, we outline practical defenses against such attacks. Code to reproduce our experiments is available at https://github.com/neulab/RIPPLe.

Key facts. Cited by 318 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 40; 2023: 68; 2024: 77; 2025: 62; 2026: 35.

318 papers.

Sort:

By area:

  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  2. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  3. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  4. 2023Tool Learning with Foundation Modelscited by 302
  5. 2023Poisoning Language Models During Instruction Tuningcited by 254
  6. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  7. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  8. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfercited by 147
  9. 2024Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agentscited by 135
  10. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 127
  11. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  12. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
38 more
  1. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  2. 2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
  3. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  4. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  5. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 99
  6. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 97
  7. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 96
  8. 2023Text-to-Image Diffusion Models can be Easily Backdoored through Multimodal Data Poisoningcited by 91
  9. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  10. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  11. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 87
  12. 2021Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitutioncited by 85
  13. 2024BadEdit: Backdooring large language models by model editingcited by 84
  14. 2021Hidden Backdoors in Human-Centric Language Modelscited by 83
  15. 2022Shortcut Learning of Large Language Models in Natural Language Understandingcited by 83
  16. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 79
  17. 2021Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Modelscited by 78
  18. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  19. 2021BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Modelscited by 76
  20. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  21. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  22. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarkscited by 70
  23. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  24. 2022ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classificationcited by 66
  25. 2024Instructional Fingerprinting of Large Language Modelscited by 64
  26. 2023Composite Backdoor Attacks Against Large Language Modelscited by 63
  27. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labelscited by 61
  28. 2022Rickrolling the Artist: Injecting Backdoors into Text Encoders for Text-to-Image Synthesiscited by 59
  29. 2022BadPrompt: Backdoor Attacks on Continuous Promptscited by 59
  30. 2022Exploring the Universal Vulnerability of Prompt-based Learning Paradigmcited by 56
  31. 2023SeqXGPT: Sentence-Level AI-Generated Text Detectioncited by 55
  32. 2023Detecting Backdoors in Pre-trained Encoderscited by 53
  33. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Modelscited by 50
  34. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  35. 2022BITE: Textual Backdoor Attacks with Iterative Trigger Injectioncited by 47
  36. 2024Persistent Pre-Training Poisoning of LLMscited by 46
  37. 2022BadDet: Backdoor Attacks on Object Detectioncited by 45
  38. 2023An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registrycited by 45

50 most cited of 318 All 318 (JSON)