mlsanityData to 7 Oct 2026

Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks

arXiv 2305.14965 ↗First posted 24 May 2023cs.CLcited by 45

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Recent explorations with commercial Large Language Models (LLMs) have shown that non-expert users can jailbreak LLMs by simply manipulating their prompts; resulting in degenerate output behavior, privacy and security breaches, offensive outputs, and violations of content regulator policies. Limited studies have been conducted to formalize and analyze these attacks and their mitigations. We bridge this gap by proposing a formalism and a taxonomy of known (and possible) jailbreaks. We survey existing jailbreak methods and their effectiveness on open-source and commercial LLMs (such as GPT-based models, OPT, BLOOM, and FLAN-T5-XXL). We further discuss the challenges of jailbreak detection in terms of their effectiveness against known attacks. For further analysis, we release a dataset of model outputs across 3700 jailbreak prompts over 4 tasks.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 29 of its 58 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

29 of 58 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2022Red Teaming Language Models with Language Modelscited by 983
17 more
  1. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  2. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  3. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  4. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  5. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  6. 2023Poisoning Language Models During Instruction Tuningcited by 254
  7. 2021Latent Hatred: A Benchmark for Understanding Implicit Hate Speechcited by 216
  8. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  9. 2021Alignment of Language Agentscited by 167
  10. 2023A Categorical Archive of ChatGPT Failurescited by 167
  11. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  12. 2020A Survey of Privacy Attacks in Machine Learningcited by 143
  13. 2023Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Modelscited by 87
  14. 2021Hidden Backdoors in Human-Centric Language Modelscited by 83
  15. 2021Spinning Language Models: Risks of Propaganda-As-A-Service and Countermeasurescited by 69
  16. 2021Detecting Hate Speech with GPT-3cited by 55
  17. 2023Training-free Lexical Backdoor Attacks on Language Modelscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 11; 2025: 22; 2026: 10.

45 papers.

Sort:

By area:

  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  2. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  3. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  4. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  5. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  6. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 24
  7. 2025DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluationcited by 21
  8. 2024Jailbreaking Large Language Models with Symbolic Mathematicscited by 14
  9. 2025Adversarial Poetry as a Universal Single-Turn Jailbreak Mechanism in Large Language Modelscited by 14
  10. 2025Persuasion Dynamics in LLMs: Investigating Robustness and Adaptability in Knowledge and Safety with DuET-PDcited by 13
  11. 2023A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Modelscited by 12
  12. 2024REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Spacecited by 12
33 more
  1. 2025JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoringcited by 12
  2. 2025Episodic memory in AI agents poses risks that should be studied and mitigatedcited by 11
  3. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  4. 2024RACONTEUR: A Knowledgeable, Insightful, and Portable LLM-Powered Shell Command Explainercited by 9
  5. 2025Agents Are All You Need for LLM Unlearningcited by 9
  6. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  7. 2025Guardians and Offenders: A Survey on Harmful Content Generation and Safety Mitigation of LLMcited by 8
  8. 2025Preventing Prompt Injection with Type-Directed Privilege Separationcited by 8
  9. 2025What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMscited by 7
  10. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Samplingcited by 6
  11. 2024MARCO: Multi-Agent Real-time Chat Orchestrationcited by 5
  12. 2026ICON: Intent-Context Coupling for Efficient Multi-Turn Jailbreak Attackcited by 5
  13. 2024Non-Halting Queries: Exploiting Fixed Points in LLMscited by 4
  14. 2025Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problemcited by 4
  15. 2025On Surjectivity of Neural Networks: Can you elicit any behavior from your model?cited by 4
  16. 2025AntiDote: Bi-level Adversarial Training for Tamper-Resistant LLMscited by 4
  17. 2025Auditing Pay-Per-Token in Large Language Modelscited by 4
  18. 2025Guarding the Guardrails: A Taxonomy-Driven Approach to Jailbreak Detectioncited by 4
  19. 2026JevOut: Natural Context Can Flip Decision Modelscited by 4
  20. 2025From Adversarial Poetry to Adversarial Tales: An Interpretability Research Agendacited by 3
  21. 2026You Told Me to Do It: Measuring Instructional Text-induced Private Data Leakage in LLM Agentscited by 3
  22. 2026Adversarial Humanities Benchmark: Results on Stylistic Robustness in Frontier Model Safetycited by 3
  23. 2024[WIP] Jailbreak Paradox: The Achilles' Heel of LLMscited by 2
  24. 2026SoK: Robustness in Large Language Models against Jailbreak Attackscited by 2
  25. 2025An Audit and Analysis of LLM-Assisted Health Misinformation Jailbreaks Against LLMscited by 1
  26. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  27. 2026Membrane: A Self-Evolving Contrastive Safety Memory for LLM Agent Defensecited by 1
  28. 2025Improving LLM Outputs Against Jailbreak Attacks with Expert Model Integrationcited by 0
  29. 2025Breaking to Build: A Threat Model of Prompt-Based Attacks for Securing LLMscited by 0
  30. 2026Metaphor Is Not All Attention Needscited by 0
  31. 2026EVA: Editing for Versatile Alignment against Jailbreakscited by 0
  32. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Modelscited by 0
  33. 2026SAFEGuard: Detect Optimization-Based Jailbreak Attacks Through Harmful Semantic Analysis and Fluency Measurementcited by 0