mlsanityData to 7 Oct 2026

Jailbroken: How Does LLM Safety Training Fail?

arXiv 2307.02483 ↗First posted 5 Jul 2023cs.LG cs.CRcited by 1,565

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models trained for safety and harmlessness remain susceptible to adversarial misuse, as evidenced by the prevalence of "jailbreak" attacks on early releases of ChatGPT that elicit undesired behavior. Going beyond recognition of the issue, we investigate why such attacks succeed and how they can be created. We hypothesize two failure modes of safety training: competing objectives and mismatched generalization. Competing objectives arise when a model's capabilities and safety goals conflict, while mismatched generalization occurs when safety training fails to generalize to a domain for which capabilities exist. We use these failure modes to guide jailbreak design and then evaluate state-of-the-art models, including OpenAI's GPT-4 and Anthropic's Claude v1.3, against both existing and newly designed attacks. We find that vulnerabilities persist despite the extensive red-teaming and safety-training efforts behind these models. Notably, new attacks utilizing our failure modes succeed on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets and outperform existing ad hoc jailbreaks. Our analysis emphasizes the need for safety-capability parity -- that safety mechanisms should be as sophisticated as the underlying model -- and argues against the idea that scaling alone can resolve these safety failure modes.

Key facts. Cited by 1,565 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 30 of its 67 references that are in the map. First posted on arXiv 5 Jul 2023.

Builds on

30 of 67 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2022Emergent Abilities of Large Language Modelscited by 2,483
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
18 more
  1. 2022Red Teaming Language Models with Language Modelscited by 983
  2. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  5. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  6. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  7. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  8. 2018Adversarial Attacks and Defences: A Surveycited by 259
  9. 2023Pretraining Language Models with Human Preferencescited by 245
  10. 2022On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoningcited by 191
  11. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  12. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  13. 2021Challenges in Detoxifying Language Modelscited by 169
  14. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  15. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 157
  16. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  17. 2022Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Modelscited by 67
  18. 2022On the Impossible Safety of Large AI Modelscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 100; 2024: 467; 2025: 509; 2026: 489.

1,565 papers.

Sort:

By area:

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  7. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  8. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  10. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  12. 2023A Comprehensive Overview of Large Language Modelscited by 639
38 more
  1. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  3. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  6. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 478
  7. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  8. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  9. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  10. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  11. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  12. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  13. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  14. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  15. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
  16. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  17. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  18. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  19. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  20. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  21. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  22. 2024A StrongREJECT for Empty Jailbreakscited by 352
  23. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  24. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  25. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  26. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  27. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  28. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  29. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 280
  30. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  31. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  32. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  33. 2023Challenges and Applications of Large Language Modelscited by 251
  34. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  35. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  36. 2023Certifying LLM Safety against Adversarial Promptingcited by 248
  37. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  38. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247

50 most cited of 1,565 All 1,565 (JSON)