mlsanityData to 7 Oct 2026

Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks

arXiv 2404.02151 ↗First posted 2 Apr 2024cs.CR cs.AI cs.LGcited by 450

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

We show that even the most recent safety-aligned LLMs are not robust to simple adaptive jailbreaking attacks. First, we demonstrate how to successfully leverage access to logprobs for jailbreaking: we initially design an adversarial prompt template (sometimes adapted to the target LLM), and then we apply random search on a suffix to maximize a target logprob (e.g., of the token "Sure"), potentially with multiple restarts. In this way, we achieve 100% attack success rate -- according to GPT-4 as a judge -- on Vicuna-13B, Mistral-7B, Phi-3-Mini, Nemotron-4-340B, Llama-2-Chat-7B/13B/70B, Llama-3-Instruct-8B, Gemma-7B, GPT-3.5, GPT-4o, and R2D2 from HarmBench that was adversarially trained against the GCG attack. We also show how to jailbreak all Claude models -- that do not expose logprobs -- via either a transfer or prefilling attack with a 100% success rate. In addition, we show how to use random search on a restricted set of tokens for finding trojan strings in poisoned models -- a task that shares many similarities with jailbreaking -- which is the algorithm that brought us the first place in the SaTML'24 Trojan Detection Competition. The common theme behind these attacks is that adaptivity is crucial: different models are vulnerable to different prompting templates (e.g., R2D2 is very sensitive to in-context learning prompts), some models have unique vulnerabilities based on their APIs (e.g., prefilling for Claude), and in some settings, it is crucial to restrict the token search space based on prior knowledge (e.g., for trojan detection). For reproducibility purposes, we provide the code, logs, and jailbreak artifacts in the JailbreakBench format at https://github.com/tml-epfl/llm-adaptive-attacks.

Key facts. Cited by 450 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 42 of its 65 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

42 of 65 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  2. 2024PRP: Propagating Universal Perturbations to Attack Large Language Model Guard-Rails24 Feb
  3. 2024Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology24 Feb
  4. 2024Query-Based Adversarial Prompt Generation19 Feb
  5. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks14 Feb
  6. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  7. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks18 Jan
  8. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  9. 2023Bypassing the Safety Training of Open-Source LLMs with Priming Attacks19 Dec
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  11. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
7 more
  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  2. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  4. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  5. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  7. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Mistral 7Bcited by 3,065
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  12. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
11 more
  1. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  3. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  4. 2019On Evaluating Adversarial Robustnesscited by 442
  5. 2017Wild Patterns: Ten Years After the Rise of Adversarial Machine Learningcited by 411
  6. 2018Certified Defenses against Adversarial Examplescited by 305
  7. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  8. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  9. 2020Sparse-RS: a versatile framework for query-efficient sparse black-box adversarial attackscited by 65
  10. 2022Evaluating the Adversarial Robustness of Adaptive Test-time Defensescited by 64
  11. 2024PAL: Proxy-Guided Black-Box Attack on Large Language Modelscited by 46

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 100; 2025: 204; 2026: 146.

450 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  3. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 406
  4. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  5. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  6. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  7. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  8. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  9. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  10. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  11. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  12. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
38 more
  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  2. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  3. 2025OS-Harm: A Benchmark for Measuring Safety of Computer Use Agentscited by 104
  4. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  5. 2025OverThink: Slowdown Attacks on Reasoning LLMscited by 94
  6. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  7. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 74
  8. 2024Best-of-N Jailbreakingcited by 71
  9. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  10. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  11. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  12. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  13. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  14. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  15. 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 60
  16. 2024SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Mannercited by 57
  17. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Timecited by 57
  18. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  19. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 53
  20. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  21. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  22. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  23. 2024MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Controlcited by 49
  24. 2025Commercial LLM Agents Are Already Vulnerable to Simple Yet Dangerous Attackscited by 49
  25. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 49
  26. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  27. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  28. 2024Steering Without Side Effects: Improving Post-Deployment Control of Language Modelscited by 44
  29. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  30. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  31. 2024Robustifying Safety-Aligned Large Language Models through Clean Data Curationcited by 43
  32. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  33. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  34. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Testercited by 41
  35. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  36. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  37. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  38. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35

50 most cited of 450 All 450 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Paper
  2. 2025Proactive defense against LLM Jailbreak
  3. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  4. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs
  5. 2024Break the Breakout: Reinventing LM Defense Against Jailbreak Attacks with Self-Refinement
  6. 2024Improved Generation of Adversarial Examples Against Safety-aligned LLMs
  7. 2026Circuit Discovery Helps Detect LLM Jailbreaking: A Mechanistic Interpretability Study
  8. 2025AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defender