mlsanityData to 14 Oct 2026

GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts

arXiv 2309.10253 ↗First posted 19 Sep 2023cs.AIcited by 560

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) have recently experienced tremendous popularity and are widely used from casual conversations to AI-driven programming. However, despite their considerable success, LLMs are not entirely reliable and can give detailed guidance on how to conduct harmful or illegal activities. While safety measures can reduce the risk of such outputs, adversarial jailbreak attacks can still exploit LLMs to produce harmful content. These jailbreak templates are typically manually crafted, making large-scale testing challenging. In this paper, we introduce GPTFuzz, a novel black-box jailbreak fuzzing framework inspired by the AFL fuzzing framework. Instead of manual engineering, GPTFuzz automates the generation of jailbreak templates for red-teaming LLMs. At its core, GPTFuzz starts with human-written templates as initial seeds, then mutates them to produce new templates. We detail three key components of GPTFuzz: a seed selection strategy for balancing efficiency and variability, mutate operators for creating semantically equivalent or similar sentences, and a judgment model to assess the success of a jailbreak attack. We evaluate GPTFuzz against various commercial and open-source LLMs, including ChatGPT, LLaMa-2, and Vicuna, under diverse attack scenarios. Our results indicate that GPTFuzz consistently produces jailbreak templates with a high success rate, surpassing human-crafted templates. Remarkably, GPTFuzz achieves over 90% attack success rates against ChatGPT and Llama-2 models, even with suboptimal initial seed templates. We anticipate that GPTFuzz will be instrumental for researchers and practitioners in examining LLM robustness and will encourage further exploration into enhancing LLM safety.

Key facts. Cited by 560 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 45 of its 78 references that are in the map. First posted on arXiv 19 Sep 2023.

Builds on

45 of 78 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  3. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  4. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2017On Calibration of Modern Neural Networkscited by 4,960
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
29 more
  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  4. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  5. 2023PaLM 2 Technical Reportcited by 1,162
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  7. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  8. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  9. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  10. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  11. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  12. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  13. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  14. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 333
  15. 2023How is ChatGPT's behavior changing over time?cited by 270
  16. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  17. 2021Challenges in Detoxifying Language Modelscited by 169
  18. 2020Adversarial Training for Large Neural Language Modelscited by 111
  19. 2021Evaluating the Robustness of Neural Language Models to Input Perturbationscited by 87
  20. 2023Safety Assessment of Chinese Large Language Modelscited by 85
  21. 2023CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibilitycited by 85
  22. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 73
  23. 2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Modelscited by 57
  24. 2023NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Modelscited by 37
  25. 2023Multi-target Backdoor Attacks for Code Pre-trained Modelscited by 23
  26. 2021Snipuzz: Black-box Fuzzing of IoT Firmware via Message Snippet Inferencecited by 6
  27. 2021Evolutionary Mutation-based Fuzzing as Monte Carlo Tree Searchcited by 2
  28. 2021DIAR: Removing Uninteresting Bytes from Seeds in Software Fuzzingcited by 2
  29. 2022Efficient Greybox Fuzzing to Detect Memory Errorscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 20; 2024: 199; 2025: 212; 2026: 129.

560 papers.

Sort:

By area:

  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  2. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  3. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  4. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  5. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  6. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  7. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  8. 2024A StrongREJECT for Empty Jailbreakscited by 385
  9. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  10. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 247
  11. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  12. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
38 more
  1. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 216
  2. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  3. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  4. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  5. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 172
  6. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 170
  7. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 156
  8. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  9. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 150
  10. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  11. 2024The Emerged Security and Privacy of LLM Agent: A Survey with Case Studiescited by 131
  12. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  13. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 126
  14. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  15. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  16. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  17. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 111
  18. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  19. 2024Jailbreak Vision Language Models via Bi-Modal Adversarial Promptcited by 109
  20. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  21. 2024Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstructioncited by 107
  22. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 106
  23. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  24. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 102
  25. 2024DrAttack: Prompt Decomposition and Reconstruction Makes Powerful LLM Jailbreakerscited by 101
  26. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 98
  27. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 96
  28. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 95
  29. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  30. 2024Coercing LLMs to do and reveal (almost) anythingcited by 89
  31. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 88
  32. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  33. 2024Detoxifying Large Language Models via Knowledge Editingcited by 87
  34. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 86
  35. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
  36. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 83
  37. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 81
  38. 2024PsySafe: A Comprehensive Framework for Psychological-based Attack, Defense, and Evaluation of Multi-agent System Safetycited by 80

50 most cited of 560 All 560 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily
  2. 2025Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models
  3. 2023FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models
  4. 2025Advancing Jailbreak Strategies: A Hybrid Approach to Exploiting LLM Vulnerabilities and Bypassing Modern Defenses
  5. 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
  6. 2025AutoAdv: Automated Adversarial Prompting for Multi-Turn Jailbreaking of Large Language Models
  7. 2026The Art of the Jailbreak: Formulating Jailbreak Attacks for LLM Security Beyond Binary Scoring
  8. 2024PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMs