mlsanityData to 7 Oct 2026

Defending Against Backdoor Attacks in Natural Language Generation

arXiv 2106.01810 ↗First posted 3 Jun 2021cs.CLcited by 30

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

The frustratingly fragile nature of neural network models make current natural language generation (NLG) systems prone to backdoor attacks and generate malicious sequences that could be sexist or offensive. Unfortunately, little effort has been invested to how backdoor attacks can affect current NLG models and how to defend against these attacks. In this work, by giving a formal definition of backdoor attack and defense, we investigate this problem on two important NLG tasks, machine translation and dialog generation. Tailored to the inherent nature of NLG models (e.g., producing a sequence of coherent words given contexts), we design defending strategies against attacks. We find that testing the backward probability of generating sources given targets yields effective defense performance against all different types of attacks, and is able to handle the it one-to-many issue in many NLG tasks such as dialog generation. We hope that this work can raise the awareness of backdoor risks concealed in deep NLG systems and inspire more future work (both attack and defense) towards this direction.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 64 of its 96 references that are in the map. First posted on arXiv 3 Jun 2021.

Builds on

64 of 96 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May
  2. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models29 Mar

Other references · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
38 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  5. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  6. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  7. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  8. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  9. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  10. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  12. 2017Convolutional Sequence to Sequence Learningcited by 550
  13. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  14. 2019Hidden Trigger Backdoor Attackscited by 359
  15. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  16. 2020Input-Aware Dynamic Backdoor Attackcited by 304
  17. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  18. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  19. 2019FreeLB: Enhanced Adversarial Training for Natural Language Understandingcited by 261
  20. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  21. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  22. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  23. 2015A Neural Conversational Modelcited by 195
  24. 2017Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimizationcited by 188
  25. 2018Neural Approaches to Conversational AIcited by 164
  26. 2020Witches' Brew: Industrial Scale Data Poisoning via Gradient Matchingcited by 156
  27. 2019TABOR: A Highly Accurate Approach to Inspecting and Restoring Trojan Backdoors in AI Systemscited by 120
  28. 2020MetaPoison: Practical General-purpose Clean-label Data Poisoningcited by 114
  29. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  30. 2020Just How Toxic is Data Poisoning? A Unified Benchmark for Backdoor and Data Poisoning Attackscited by 109
  31. 2017Deep Text Classification Can be Fooledcited by 105
  32. 2017Adversarial Learning for Neural Dialogue Generationcited by 103
  33. 2016Crafting Adversarial Input Sequences for Recurrent Neural Networkscited by 102
  34. 2019Defending Neural Backdoors via Generative Distribution Modelingcited by 82
  35. 2020Mitigating backdoor attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identificationcited by 80
  36. 2020Rethinking the Trigger of Backdoor Attackcited by 76
  37. 2018Interpretable Adversarial Perturbation in Input Embedding Space for Textcited by 67
  38. 2020Stealthy and Efficient Adversarial Attacks against Deep Reinforcement Learningcited by 62

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 5; 2024: 12; 2025: 7; 2026: 2.

30 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  3. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 83
  4. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labelscited by 61
  5. 2023A Pathway Towards Responsible AI Generated Contentcited by 38
  6. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  7. 2024Test-Time Backdoor Attacks on Multimodal Large Language Modelscited by 35
  8. 2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 34
  9. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  10. 2022A Survey of Adversarial Defences and Robustness in NLPcited by 22
  11. 2024BadActs: A Universal Backdoor Defense in the Activation Spacecited by 19
  12. 2024Defending Against Weight-Poisoning Backdoor Attacks for Parameter-Efficient Fine-Tuningcited by 16
18 more
  1. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 15
  2. 2022A Survey on Backdoor Attack and Defense in Natural Language Processingcited by 13
  3. 2024LLMScan: Causal Scan for LLM Misbehavior Detectioncited by 13
  4. 2025BadJudge: Backdoor Vulnerabilities of LLM-as-a-Judgecited by 13
  5. 2023Backdoor Attacks with Input-unique Triggers in NLPcited by 10
  6. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputscited by 10
  7. 2024Backdoor Attack on Multilingual Machine Translationcited by 5
  8. 2024Securing Multi-turn Conversational Language Models From Distributed Backdoor Triggerscited by 5
  9. 2023Survey on AI Ethics: A Socio-technical Perspectivecited by 4
  10. 2025Exploring Backdoor Attack and Defense for LLM-empowered Recommendationscited by 4
  11. 2025Paladin: Defending LLM-enabled Phishing Emails with a New Trigger-Tag Paradigmcited by 4
  12. 2025Forgetting to Forget: Attention Sink as A Gateway for Backdooring LLM Unlearningcited by 4
  13. 2024Augmented Neural Fine-Tuning for Efficient Backdoor Purificationcited by 3
  14. 2025Neuroplasticity and Corruption in Model Mechanisms: A Case Study Of Indirect Object Identificationcited by 3
  15. 2024Fisher Information guided Purification against Backdoor Attackscited by 2
  16. 2026Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Modelscited by 2
  17. 2021A General Framework for Defending Against Backdoor Attacks via Influence Graphcited by 1
  18. 2026STEP: Detecting Audio Backdoor Attacks via Stability-based Trigger Exposure Profilingcited by 0