mlsanityData to 7 Oct 2026

Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study

arXiv 2305.13860 ↗First posted 23 May 2023cs.SE cs.AI cs.CLcited by 478

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large Language Models (LLMs), like ChatGPT, have demonstrated vast potential but also introduce challenges related to content constraints and potential misuse. Our study investigates three key research questions: (1) the number of different prompt types that can jailbreak LLMs, (2) the effectiveness of jailbreak prompts in circumventing LLM constraints, and (3) the resilience of ChatGPT against these jailbreak prompts. Initially, we develop a classification model to analyze the distribution of existing prompts, identifying ten distinct patterns and three categories of jailbreak prompts. Subsequently, we assess the jailbreak capability of prompts with ChatGPT versions 3.5 and 4.0, utilizing a dataset of 3,120 jailbreak questions across eight prohibited scenarios. Finally, we evaluate the resistance of ChatGPT against jailbreak prompts, finding that the prompts can consistently evade the restrictions in 40 use-case scenarios. The study underscores the importance of prompt structures in jailbreaking LLMs and discusses the challenges of robust jailbreak prompt generation and prevention.

Key facts. Cited by 478 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 9 of its 22 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

9 of 22 references.

References · 9

  1. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  2. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  4. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  5. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  6. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  7. 2023A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?cited by 84
  8. 2023Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineeringcited by 40
  9. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 61; 2024: 213; 2025: 125; 2026: 79.

478 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  5. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  6. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  7. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  8. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  9. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  11. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  12. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
38 more
  1. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  2. 2024A StrongREJECT for Empty Jailbreakscited by 352
  3. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  4. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  5. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  6. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  7. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  8. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  9. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  10. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  11. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  12. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  13. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  14. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  15. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  16. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  17. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  18. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  19. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 189
  20. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  21. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  22. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  23. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  24. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  25. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  26. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  27. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  28. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  29. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  30. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 128
  31. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  32. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  33. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  34. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  35. 2026SoK: Agentic Skills -- Beyond Tool Use in LLM Agentscited by 120
  36. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  37. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  38. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110

50 most cited of 478 All 478 (JSON)