mlsanityData to 7 Oct 2026

"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

arXiv 2308.03825 ↗First posted 7 Aug 2023cs.CR cs.LGcited by 457

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The misuse of large language models (LLMs) has drawn significant attention from the general public and LLM vendors. One particular type of adversarial prompt, known as jailbreak prompt, has emerged as the main attack vector to bypass the safeguards and elicit harmful content from LLMs. In this paper, employing our new framework JailbreakHub, we conduct a comprehensive analysis of 1,405 jailbreak prompts spanning from December 2022 to December 2023. We identify 131 jailbreak communities and discover unique characteristics of jailbreak prompts and their major attack strategies, such as prompt injection and privilege escalation. We also observe that jailbreak prompts increasingly shift from online Web communities to prompt-aggregation websites and 28 user accounts have consistently optimized jailbreak prompts over 100 days. To assess the potential harm caused by jailbreak prompts, we create a question set comprising 107,250 samples across 13 forbidden scenarios. Leveraging this dataset, our experiments on six popular LLMs show that their safeguards cannot adequately defend jailbreak prompts in all scenarios. Particularly, we identify five highly effective jailbreak prompts that achieve 0.95 attack success rates on ChatGPT (GPT-3.5) and GPT-4, and the earliest one has persisted online for over 240 days. We hope that our study can facilitate the research community and LLM vendors in promoting safer and regulated LLMs.

Key facts. Cited by 457 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 48 of its 97 references that are in the map. First posted on arXiv 7 Aug 2023.

Builds on

48 of 97 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  3. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  11. 2023PaLM 2 Technical Reportcited by 1,154
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
33 more
  1. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  4. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  5. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 663
  6. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  7. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  8. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  9. 2020The Pushshift Reddit Datasetcited by 289
  10. 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 282
  11. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  12. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  13. 2019Detecting AI Trojans Using Meta Neural Analysiscited by 192
  14. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  15. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  16. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
  17. 2023On the Reliability of Watermarks for Large Language Modelscited by 170
  18. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  19. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  20. 2023MGTBench: Benchmarking Machine-Generated Text Detectioncited by 90
  21. 2023Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Taskscited by 88
  22. 2021Bad Characters: Imperceptible NLP Attackscited by 69
  23. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  24. 2021PalmTree: Learning an Assembly Language Model for Instruction Embeddingcited by 49
  25. 2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 48
  26. 2023In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPTcited by 35
  27. 2022Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbotscited by 31
  28. 2023WizMap: Scalable Interactive Visualization for Exploring Large Machine Learning Embeddingscited by 28
  29. 2018A Quantitative Approach to Understanding Online Antisemitismcited by 26
  30. 2023To ChatGPT, or not to ChatGPT: That is the question!cited by 20
  31. 2019"And We Will Fight For Our Race!" A Measurement Study of Genetic Testing Conversations on Reddit and 4chancited by 10
  32. 2022On Xing Tian and the Perseverance of Anti-China Sentiment Onlinecited by 10
  33. 2022PLUE: Language Understanding Evaluation Benchmark for Privacy Policies in Englishcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 29; 2024: 132; 2025: 167; 2026: 129.

457 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024OpenAI o1 System Cardcited by 1,220
  3. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  6. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  7. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  8. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  9. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  10. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  11. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  12. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
38 more
  1. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  2. 2024A StrongREJECT for Empty Jailbreakscited by 352
  3. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  4. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  6. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  7. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  8. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  9. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  10. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  11. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  12. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  13. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLMcited by 173
  14. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  15. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  16. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  17. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  18. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  19. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  20. 2023On the Adversarial Robustness of Multi-Modal Foundation Modelscited by 142
  21. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  22. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  23. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  24. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  25. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 121
  26. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Modelscited by 119
  27. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  28. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  29. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  30. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  31. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  32. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  33. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  34. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  35. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  36. 2025Persona Features Control Emergent Misalignmentcited by 76
  37. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  38. 2024Safeguarding Large Language Models: A Surveycited by 73

50 most cited of 457 All 457 (JSON)