mlsanityData to 7 Oct 2026

Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models

arXiv 2403.17336 ↗First posted 26 Mar 2024cs.CR cs.CLcited by 77

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Recent advancements in generative AI have enabled ubiquitous access to large language models (LLMs). Empowered by their exceptional capabilities to understand and generate human-like text, these models are being increasingly integrated into our society. At the same time, there are also concerns on the potential misuse of this powerful technology, prompting defensive measures from service providers. To overcome such protection, jailbreaking prompts have recently emerged as one of the most effective mechanisms to circumvent security restrictions and elicit harmful content originally designed to be prohibited. Due to the rapid development of LLMs and their ease of access via natural languages, the frontline of jailbreak prompts is largely seen in online forums and among hobbyists. To gain a better understanding of the threat landscape of semantically meaningful jailbreak prompts, we systemized existing prompts and measured their jailbreak effectiveness empirically. Further, we conducted a user study involving 92 participants with diverse backgrounds to unveil the process of manually creating jailbreak prompts. We observed that users often succeeded in jailbreak prompts generation regardless of their expertise in LLMs. Building on the insights from the user study, we also developed a system using AI as the assistant to automate the process of jailbreak prompt generation.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 28 of its 60 references that are in the map. First posted on arXiv 26 Mar 2024.

Builds on

28 of 60 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  2. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2023PaLM 2 Technical Reportcited by 1,154
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 911
  12. 2021Membership Inference Attacks From First Principlescited by 839
14 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  3. 2020Weight Poisoning Attacks on Pre-trained Modelscited by 318
  4. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  5. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  6. 2023Poisoning Language Models During Instruction Tuningcited by 254
  7. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  8. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  9. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  10. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  11. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  12. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  13. 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 112
  14. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 27; 2026: 25.

77 papers.

Sort:

By area:

  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  2. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  3. 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 58
  4. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  5. 2024ALI-Agent: Assessing LLMs' Alignment with Human Values via Agent-based Evaluationcited by 46
  6. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
  7. 2024Talking Heads: Understanding Inter-layer Communication in Transformer Language Modelscited by 39
  8. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agentcited by 39
  9. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  10. 2024Unleashing Worms and Extracting Data: Escalating the Outcome of Attacks against RAG-based Inference in Scale and Severity Using Jailbreakingcited by 30
  11. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  12. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 28
38 more
  1. 2024Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trendscited by 26
  2. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  3. 2024h4rm3l: A language for Composable Jailbreak Attack Synthesiscited by 23
  4. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  5. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 22
  6. 2025MCPXKIT: The Unified Toolkit for Analyzing Model Context Protocol Securitycited by 21
  7. 2024PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMscited by 20
  8. 2025NeuroStrike: Neuron-Level Attacks on Aligned LLMscited by 18
  9. 2025T2VShield: Model-Agnostic Jailbreak Defense for Text-to-Video Modelscited by 17
  10. 2025Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Modelscited by 14
  11. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 13
  12. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  13. 2025From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agentscited by 10
  14. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
  15. 2025Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Modelscited by 9
  16. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  17. 2024Adversarial Vulnerabilities in Large Language Models for Time Series Forecastingcited by 6
  18. 2026JailNewsBench: Multi-Lingual and Regional Benchmark for Fake News Generation under Jailbreak Attackscited by 6
  19. 2024LLM Safeguard is a Double-Edged Sword: Exploiting False Positives for Denial-of-Service Attackscited by 5
  20. 2024Are You Human? An Adversarial Benchmark to Expose LLMscited by 5
  21. 2025LLMZ+: Contextual Prompt Whitelist Principles for Agentic LLMscited by 5
  22. 2024Towards Action Hijacking of Large Language Model-based Agentcited by 4
  23. 2025Geneshift: Impact of different scenario shift on Jailbreaking LLMcited by 4
  24. 2025Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detectioncited by 4
  25. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  26. 2026Towards Security-Auditable LLM Agents: A Unified Graph Representationcited by 4
  27. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMscited by 3
  28. 2025XBreaking: Understanding how LLMs security alignment can be brokencited by 3
  29. 2025Beyond Jailbreaks: Revealing Stealthier and Broader LLM Security Risks Stemming from Alignment Failurescited by 3
  30. 2026Measuring Pragmatic Influence in Large Language Model Instructionscited by 3
  31. 2026Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safetycited by 3
  32. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  33. 2025Jailbreaking LLMs via Semantically Relevant Nested Scenarios with Targeted Toxic Knowledgecited by 2
  34. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  35. 2024Quantifying Risk Propensities of Large Language Models: Ethical Focus and Bias Detection through Role-Playcited by 1
  36. 2024Divide and Conquer: A Hybrid Strategy Defeats Multimodal Large Language Modelscited by 1
  37. 2025Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTscited by 1
  38. 2025AC-LoRA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMscited by 1

50 most cited of 77 All 77 (JSON)