mlsanityData to 7 Oct 2026

BadGPT: Exploring Security Vulnerabilities of ChatGPT via Backdoor Attacks to InstructGPT

arXiv 2304.12298 ↗First posted 21 Feb 2023cs.CR cs.AIcited by 70

On the map: LLM safety and security → Backdoor and data poisoning attacks

Abstract (arXiv)

Recently, ChatGPT has gained significant attention in research due to its ability to interact with humans effectively. The core idea behind this model is reinforcement learning (RL) fine-tuning, a new paradigm that allows language models to align with human preferences, i.e., InstructGPT. In this study, we propose BadGPT, the first backdoor attack against RL fine-tuning in language models. By injecting a backdoor into the reward model, the language model can be compromised during the fine-tuning stage. Our initial experiments on movie reviews, i.e., IMDB, demonstrate that an attacker can manipulate the generated text through BadGPT.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Backdoor and data poisoning attacks, one of 12 directions in LLM safety and security. Builds on 5 of its 7 references that are in the map. First posted on arXiv 21 Feb 2023.

Builds on

5 of 7 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022BadPrompt: Backdoor Attacks on Continuous Prompts27 Nov

Other references · 4

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  3. 2023A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPTcited by 364
  4. 2022BackdoorBench: A Comprehensive Benchmark of Backdoor Learningcited by 124

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 13; 2024: 33; 2025: 15; 2026: 9.

70 papers.

Sort:

By area:

  1. 2023Large Language Model Alignment: A Surveycited by 241
  2. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  3. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  4. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  5. 2024BadAgent: Inserting and Activating Backdoor Attacks in LLM Agentscited by 106
  6. 2023Instructions as Backdoors: Backdoor Vulnerabilities of Instruction Tuning for Large Language Modelscited by 100
  7. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  8. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  9. 2024BadEdit: Backdooring large language models by model editingcited by 84
  10. 2024Safeguarding Large Language Models: A Surveycited by 73
  11. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  12. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
38 more
  1. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  2. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 50
  3. 2023Pre-trained Trojan Attacks for Visual Recognitioncited by 38
  4. 2023Beyond the Safeguards: Exploring the Security Risks of ChatGPTcited by 36
  5. 2024A Survey of Recent Backdoor Attacks and Defenses in Large Language Modelscited by 36
  6. 2024Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Datacited by 33
  7. 2023Artificial General Intelligence for Medical Imaging Analysiscited by 32
  8. 2023RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Modelscited by 32
  9. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 27
  10. 2025Security Concerns for Large Language Models: A Surveycited by 26
  11. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  12. 2024Unlearning Backdoor Attacks for LLMs with Weak-to-Strong Knowledge Distillationcited by 21
  13. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 15
  14. 2024Preference Poisoning Attacks on Reward Model Learningcited by 14
  15. 2024Has this Fact been Edited? Detecting Knowledge Edits in Language Modelscited by 14
  16. 2024Mitigating Backdoor Threats to Large Language Models: Advancement and Challengescited by 14
  17. 2024LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakagecited by 13
  18. 2025A Survey of Attacks on Large Language Modelscited by 13
  19. 2023Building Privacy-Preserving and Secure Geospatial Artificial Intelligence Foundation Modelscited by 12
  20. 2023A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Modelscited by 12
  21. 2024Trustworthy Distributed AI Systems: Robustness, Privacy, and Governancecited by 12
  22. 2023Progressive Poisoned Data Isolation for Training-time Backdoor Defensecited by 10
  23. 2024The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputscited by 10
  24. 2023An Overview on Generative AI at Scale with Edge-Cloud Computingcited by 9
  25. 2025Graph of Attacks with Pruning: Optimizing Stealthy Jailbreak Prompt Generation for Enhanced LLM Content Moderationcited by 9
  26. 2024Imperio: Language-Guided Backdoor Attacks for Arbitrary Model Controlcited by 8
  27. 2023Adversarial Fine-Tuning of Language Models: An Iterative Optimisation Approach for the Generation and Detection of Problematic Contentcited by 7
  28. 2024SABER: Model-agnostic Backdoor Attack on Chain-of-Thought in Neural Code Generationcited by 7
  29. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
  30. 2025LLM Misalignment via Adversarial RLHF Platformscited by 5
  31. 2024TrojFM: Resource-efficient Backdoor Attacks against Very Large Foundation Modelscited by 4
  32. 2024Model-Editing-Based Jailbreak against Safety-aligned Large Language Modelscited by 4
  33. 2024Foundation Models in Federated Learning: Assessing Backdoor Vulnerabilitiescited by 3
  34. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  35. 2024Securing Federated Learning against Backdoor Threats with Foundation Model Integrationcited by 3
  36. 2025Fight Fire with Fire: Defending Against Malicious RL Fine-Tuning via Reward Neutralizationcited by 3
  37. 2026Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Rewardcited by 3
  38. 2024Breaking PEFT Limitations: Leveraging Weak-to-Strong Knowledge Transfer for Backdoor Attacks in LLMscited by 2

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models
  2. 2024Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models
  3. 2025P2P: A Poison-to-Poison Remedy for Reliable Backdoor Defense in LLMs
  4. 2024Learning to Poison Large Language Models for Downstream Manipulation
  5. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning
  6. 2025Early Approaches to Adversarial Fine-Tuning for Prompt Injection Defense: A 2022 Study of GPT-3 and Contemporary Models
  7. 2023Can we trust the evaluation on ChatGPT?
  8. 2025SRD: Reinforcement-Learned Semantic Perturbation for Backdoor Defense in VLMs