mlsanityData to 7 Oct 2026

Trojaning Language Models for Fun and Profit

arXiv 2008.00312 ↗First posted 1 Aug 2020cs.CR cs.CL cs.LGcited by 80

Not on the map. Why?

Cited by 49 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent years have witnessed the emergence of a new paradigm of building natural language processing (NLP) systems: general-purpose, pre-trained language models (LMs) are composed with simple downstream models and fine-tuned for a variety of NLP tasks. This paradigm shift significantly simplifies the system development cycles. However, as many LMs are provided by untrusted third parties, their lack of standardization or regulation entails profound security implications, which are largely unexplored. To bridge this gap, this work studies the security threats posed by malicious LMs to NLP systems. Specifically, we present TROJAN-LM, a new class of trojaning attacks in which maliciously crafted LMs trigger host NLP systems to malfunction in a highly predictable manner. By empirically studying three state-of-the-art LMs (BERT, GPT-2, XLNet) in a range of security-critical NLP tasks (toxic comment detection, question answering, text completion) as well as user studies on crowdsourcing platforms, we demonstrate that TROJAN-LM possesses the following properties: (i) flexibility - the adversary is able to flexibly dene logical combinations (e.g., 'and', 'or', 'xor') of arbitrary words as triggers, (ii) efficacy - the host systems misbehave as desired by the adversary with high probability when trigger-embedded inputs are present, (iii) specificity - the trojan LMs function indistinguishably from their benign counterparts on clean inputs, and (iv) fluency - the trigger-embedded inputs appear as fluent natural language and highly relevant to their surrounding contexts. We provide analytical justification for the practicality of TROJAN-LM, and further discuss potential countermeasures and their challenges, which lead to several promising research directions.

Key facts. Cited by 80 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 14; 2022: 10; 2023: 22; 2024: 22; 2025: 9; 2026: 3.

80 papers.

Sort:

By area:

  1. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  2. 2021Unsolved Problems in ML Safetycited by 291
  3. 2023An Overview of Catastrophic AI Riskscited by 214
  4. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Triggercited by 203
  5. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  6. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 127
  7. 2021BadEncoder: Backdoor Attacks to Pre-trained Encoders in Self-Supervised Learningcited by 120
  8. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Modelscited by 97
  9. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Modelscited by 96
  10. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 93
  11. 2023TrojDiff: Trojan Attacks on Diffusion Models with Diverse Targetscited by 88
  12. 2021Hidden Backdoors in Human-Centric Language Modelscited by 83
38 more
  1. 2021Backdoor Pre-trained Models Can Transfer to Allcited by 79
  2. 2021BadPre: Task-agnostic Backdoor Attacks to Pre-trained NLP Foundation Modelscited by 76
  3. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Modelscited by 74
  4. 2022A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarkscited by 70
  5. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  6. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  7. 2023TrojanPuzzle: Covertly Poisoning Code-Suggestion Modelscited by 47
  8. 2021Red Alarm for Pre-trained Models: Universal Vulnerability to Neuron-Level Backdoor Attackscited by 44
  9. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  10. 2023Pre-trained Trojan Attacks for Visual Recognitioncited by 38
  11. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  12. 2023Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attackscited by 32
  13. 2023Attention-Enhancing Backdoor Attacks Against BERT-based Modelscited by 32
  14. 2024Risks from Language Models for Automated Mental Healthcare: Ethics and Structure for Implementationcited by 27
  15. 2023TrojText: Test-time Invisible Textual Trojan Insertioncited by 26
  16. 2023From Shortcuts to Triggers: Backdoor Defense with Denoised PoEcited by 26
  17. 2022Kallima: A Clean-label Framework for Textual Backdoor Attackscited by 22
  18. 2023Analyzing And Editing Inner Mechanisms Of Backdoored Language Modelscited by 22
  19. 2023Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspectivecited by 19
  20. 2023Stealing the Decoding Algorithms of Language Modelscited by 19
  21. 2022Expose Backdoors on the Way: A Feature-Based Efficient Defense against Textual Backdoor Attackscited by 18
  22. 2021BFClass: A Backdoor-free Text Classification Frameworkcited by 17
  23. 2023ParaFuzz: An Interpretability-Driven Technique for Detecting Poisoned Samples in NLPcited by 16
  24. 2022A Survey on Backdoor Attack and Defense in Natural Language Processingcited by 13
  25. 2023BDMMT: Backdoor Sample Detection for Language Models through Model Mutation Testingcited by 13
  26. 2022Backdoor Vulnerabilities in Normally Trained Deep Learning Modelscited by 12
  27. 2024A Large-Scale Exploit Instrumentation Study of AI/ML Supply Chain Attacks in Hugging Face Modelscited by 12
  28. 2024CLIBE: Detecting Dynamic Backdoors in Transformer-based NLP Modelscited by 11
  29. 2022Watermarking Pre-trained Encoders in Contrastive Learningcited by 10
  30. 2024Exploring Vulnerabilities and Protections in Large Language Models: A Surveycited by 10
  31. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputscited by 10
  32. 2023Forcing Generative Models to Degenerate Ones: The Power of Data Poisoning Attackscited by 9
  33. 2024Turning Generative Models Degenerate: The Power of Data Poisoning Attackscited by 9
  34. 2025BadMoE: Backdooring Mixture-of-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Expertscited by 8
  35. 2025LLM Security: Vulnerabilities, Attacks, Defenses, and Countermeasurescited by 8
  36. 2025A Systematic Review of Poisoning Attacks Against Large Language Modelscited by 8
  37. 2022Private, Efficient, and Accurate: Protecting Models Trained by Multi-party Learning with Differential Privacycited by 7
  38. 2023FreeEagle: Detecting Complex Neural Trojans in Data-Free Casescited by 7

50 most cited of 80 All 80 (JSON)