mlsanityData to 7 Oct 2026

Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation

arXiv 2310.06987 ↗First posted 10 Oct 2023cs.CL cs.AI cs.CRcited by 439

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

The rapid progress in open-source large language models (LLMs) is significantly advancing AI development. Extensive efforts have been made before model release to align their behavior with human values, with the primary goal of ensuring their helpfulness and harmlessness. However, even carefully aligned models can be manipulated maliciously, leading to unintended behaviors, known as "jailbreaks". These jailbreaks are typically triggered by specific text inputs, often referred to as adversarial prompts. In this work, we propose the generation exploitation attack, an extremely simple approach that disrupts model alignment by only manipulating variations of decoding methods. By exploiting different generation strategies, including varying decoding hyper-parameters and sampling methods, we increase the misalignment rate from 0% to more than 95% across 11 language models including LLaMA2, Vicuna, Falcon, and MPT families, outperforming state-of-the-art attacks with 30× lower computational cost. Finally, we propose an effective alignment method that explores diverse generation strategies, which can reasonably reduce the misalignment rate under our attack. Altogether, our study underscores a major failure in current safety evaluation and alignment procedures for open-source LLMs, strongly advocating for more comprehensive red teaming and better alignment before releasing such models. Our code is available at https://github.com/Princeton-SysML/JailbreakLLM.

Key facts. Cited by 439 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 35 of its 52 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

35 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  2. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  4. 2023Latent Jailbreak: A Benchmark for Evaluating Text Safety and Output Robustness of Large Language Models17 Jul
  5. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  6. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2018Hierarchical Neural Story Generationcited by 1,182
17 more
  1. 2023PaLM 2 Technical Reportcited by 1,154
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  3. 2023LIMA: Less Is More for Alignmentcited by 1,080
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2022Red Teaming Language Models with Language Modelscited by 983
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  8. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  9. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  10. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  11. 2023Are aligned neural networks adversarially aligned?cited by 305
  12. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 268
  13. 2023Pretraining Language Models with Human Preferencescited by 245
  14. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  15. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  16. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  17. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 179; 2025: 146; 2026: 102.

439 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  6. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  7. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Promptscited by 368
  8. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  9. 2024A StrongREJECT for Empty Jailbreakscited by 352
  10. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 255
  11. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  12. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
38 more
  1. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  2. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  3. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  4. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  5. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  6. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  7. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  8. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  9. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  10. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  11. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  12. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  13. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  14. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  15. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  16. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  17. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  18. 2024Automatic and Universal Prompt Injection Attacks against Large Language Modelscited by 112
  19. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  20. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  21. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  22. 2023Evil Geniuses: Delving into the Safety of LLM-based Agentscited by 103
  23. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
  24. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  25. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 95
  26. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  27. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  28. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  29. 2024JailbreakRadar: Comprehensive Assessment of Jailbreak Attacks Against LLMscited by 87
  30. 2024Detoxifying Large Language Models via Knowledge Editingcited by 86
  31. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  32. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  33. 2024Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplificationcited by 80
  34. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinkingcited by 78
  35. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 78
  36. 2024ToolSword: Unveiling Safety Issues of Large Language Models in Tool Learning Across Three Stagescited by 77
  37. 2024A Closer Look at the Limitations of Instruction Tuningcited by 76
  38. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75

50 most cited of 439 All 439 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Defending against Jailbreak through Early Exit Generation of Large Language Models
  2. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM
  3. 2024Jailbreaking Large Language Models Through Alignment Vulnerabilities in Out-of-Distribution Settings
  4. 2025ForgeDAN: An Evolutionary Framework for Jailbreaking Aligned Large Language Models
  5. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  6. 2025HAMSA: Hijacking Aligned Compact Models via Stealthy Automation
  7. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
  8. 2025Align to Misalign: Automatic LLM Jailbreak with Meta-Optimized LLM Judges