mlsanityData to 7 Oct 2026

ChatGPT as an Attack Tool: Stealthy Textual Backdoor Attack via Blackbox Generative Model Trigger

arXiv 2304.14475 ↗First posted 27 Apr 2023cs.CR cs.LGcited by 29

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Textual backdoor attacks pose a practical threat to existing systems, as they can compromise the model by inserting imperceptible triggers into inputs and manipulating labels in the training dataset. With cutting-edge generative models such as GPT-4 pushing rewriting to extraordinary levels, such attacks are becoming even harder to detect. We conduct a comprehensive investigation of the role of black-box generative models as a backdoor attack tool, highlighting the importance of researching relative defense strategies. In this paper, we reveal that the proposed generative model-based attack, BGMAttack, could effectively deceive textual classifiers. Compared with the traditional attack methods, BGMAttack makes the backdoor trigger less conspicuous by leveraging state-of-the-art generative models. Our extensive evaluation of attack effectiveness across five datasets, complemented by three distinct human cognition assessments, reveals that Figure 4 achieves comparable attack performance while maintaining superior stealthiness relative to baseline methods.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 44 of its 62 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

44 of 62 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022A Survey on Backdoor Attack and Defense in Natural Language Processing22 Nov
  2. 2022Kallima: A Clean-label Framework for Textual Backdoor Attacks3 Jun
  3. 2021Triggerless Backdoor Attack for NLP Tasks with Clean Labels15 Nov
  4. 2021RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models15 Oct
  5. 2021Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer14 Oct
  6. 2021BFClass: A Backdoor-free Text Classification Framework22 Sep
  7. 2021Turn the Combination Lock: Learnable Textual Backdoor Attacks via Word Substitution11 Jun
  8. 2021Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger26 May
  9. 2021Hidden Backdoors in Human-Centric Language Models1 May
  10. 2021Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models29 Mar

Other references · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
22 more
  1. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  2. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  3. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  4. 2018Neural Network Acceptability Judgmentscited by 952
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  6. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  7. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 500
  8. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  9. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  10. 2016Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samplescited by 439
  11. 2018Generating Natural Language Adversarial Examplescited by 422
  12. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  13. 2020ONION: A Simple and Effective Defense Against Textual Backdoor Attackscited by 231
  14. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  15. 2022Is GPT-3 a Good Data Annotator?cited by 208
  16. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  17. 2020Reformulating Unsupervised Style Transfer as Paraphrase Generationcited by 181
  18. 2021Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoningcited by 103
  19. 2020OpenAttack: An Open-source Textual Adversarial Attack Toolkitcited by 84
  20. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  21. 2021Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairscited by 45
  22. 2022A Word is Worth A Thousand Dollars: Adversarial Attack on Tweets Fools Stock Predictionscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 15; 2025: 4; 2026: 2.

29 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  2. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  3. 2023On the Exploitability of Instruction Tuningcited by 110
  4. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  5. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  6. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  7. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  8. 2024A Safe Harbor for AI Evaluation and Red Teamingcited by 46
  9. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  10. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  11. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  12. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 21
17 more
  1. 2023Setting the Trap: Capturing and Defeating Backdoors in Pretrained Language Models through Honeypotscited by 16
  2. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  3. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  4. 2023Guiding LLM to Fool Itself: Automatically Manipulating Machine Reading Comprehension Shortcut Triggerscited by 10
  5. 2025BadMoE: Backdooring Mixture-of-Experts LLMs via Optimizing Routing Triggers and Infecting Dormant Expertscited by 8
  6. 2025A Systematic Review of Poisoning Attacks Against Large Language Modelscited by 8
  7. 2024To Err is AI : A Case Study Informing LLM Flaw Reporting Practicescited by 5
  8. 2024SynGhost: Invisible and Universal Task-agnostic Backdoor Attack via Syntactic Transfercited by 4
  9. 2024TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Modelscited by 4
  10. 2024Large Language Models are Good Attackers: Efficient and Stealthy Textual Backdoor Attackscited by 2
  11. 2024Breaking PEFT Limitations: Leveraging Weak-to-Strong Knowledge Transfer for Backdoor Attacks in LLMscited by 2
  12. 2025NLP Security and Ethics, in the Wildcited by 2
  13. 2026Merging Triggers, Breaking Backdoors: Defensive Poisoning for Instruction-Tuned Language Modelscited by 2
  14. 2024Defense Against Syntactic Textual Backdoor Attacks with Token Substitutioncited by 1
  15. 2026Stealthy Backdoor Attacks against LLMs Based on Natural Style Triggerscited by 1
  16. 2024Advancements in Recommender Systems: A Comprehensive Analysis Based on Data, Algorithms, and Evaluationcited by 0
  17. 2025Localizing Malicious Outputs from CodeLLMcited by 0