mlsanityData to 7 Oct 2026

Distract Large Language Models for Automatic Jailbreak Attack

arXiv 2403.08424 ↗First posted 13 Mar 2024cs.CR cs.AI cs.CLcited by 40

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Extensive efforts have been made before the public release of Large language models (LLMs) to align their behaviors with human values. However, even meticulously aligned LLMs remain vulnerable to malicious manipulations such as jailbreaking, leading to unintended behaviors. In this work, we propose a novel black-box jailbreak framework for automated red teaming of LLMs. We designed malicious content concealing and memory reframing with an iterative optimization algorithm to jailbreak LLMs, motivated by the research about the distractibility and over-confidence phenomenon of LLMs. Extensive experiments of jailbreaking both open-source and proprietary LLMs demonstrate the superiority of our framework in terms of effectiveness, scalability and transferability. We also evaluate the effectiveness of existing jailbreak defense methods against our attack and highlight the crucial need to develop more effective and practical defense strategies.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 30 of its 50 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

30 of 50 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreak7 Dec
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  3. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking16 Nov
  4. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  5. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  6. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  9. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  10. 2023Low-Resource Languages Jailbreak GPT-43 Oct
  11. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  12. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
2 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  2. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  8. 2022Red Teaming Language Models with Language Modelscited by 983
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  12. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
4 more
  1. 2023Automatically Auditing Large Language Models via Discrete Optimizationcited by 187
  2. 2021Prevent the Language Model from being Overconfident in Neural Machine Translationcited by 34
  3. 2023Improving Translation Faithfulness of Large Language Models via Augmenting Instructionscited by 28
  4. 2022Towards Improving Faithfulness in Abstractive Summarizationcited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 27; 2026: 4.

40 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  3. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  4. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  5. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 31
  6. 2025PR-Attack: Coordinated Prompt-RAG Attacks on Retrieval-Augmented Generation in Large Language Models via Bilevel Optimizationcited by 29
  7. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  8. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  9. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 22
  10. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMscited by 18
  11. 2025Attack and defense techniques in large language models: A survey and new perspectivescited by 13
  12. 2025Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviorscited by 12
28 more
  1. 2025Adversarial Tokenizationcited by 11
  2. 2025From LLMs to MLLMs to Agents: A Survey of Emerging Paradigms in Jailbreak Attacks and Defenses within LLM Ecosystemcited by 11
  3. 2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
  4. 2025Enhancing Jailbreak Attacks on LLMs via Persona Promptscited by 10
  5. 2025GuidedBench: Measuring and Mitigating the Evaluation Discrepancies of In-the-wild LLM Jailbreak Methodscited by 9
  6. 2025Exploring Jailbreak Attacks on LLMs through Intent Concealment and Diversioncited by 9
  7. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  8. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMscited by 7
  9. 2024SeqAR: Jailbreak LLMs with Sequential Auto-Generated Characterscited by 6
  10. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Samplingcited by 6
  11. 2025What Makes a Good Natural Language Prompt?cited by 6
  12. 2024Feint and Attack: Attention-Based Strategies for Jailbreaking and Protecting LLMscited by 5
  13. 2025Jailbreak Transferability Emerges from Shared Representationscited by 5
  14. 2025MOCHA: Are Code Language Models Robust Against Multi-Turn Malicious Coding Prompts?cited by 5
  15. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 4
  16. 2025NeuronTune: Fine-Grained Neuron Modulation for Balanced Safety-Utility Alignment in LLMscited by 3
  17. 2026FinSafetyBench: Evaluating LLM Safety in Real-World Financial Scenarioscited by 3
  18. 2024On the Validity of Traditional Vulnerability Scoring Systems for Adversarial Attacks against LLMscited by 2
  19. 2025ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Momentscited by 2
  20. 2025Assessing Socio-Cultural Alignment and Technical Safety of Sovereign LLMscited by 2
  21. 2025Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?cited by 2
  22. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  23. 2025Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoningcited by 1
  24. 2025Layer-Wise Perturbations via Sparse Autoencoders for Adversarial Text Generationcited by 1
  25. 2026Modeling LLM Unlearning as an Asymmetric Two-Task Learning Problemcited by 1
  26. 2025TombRaider: Entering the Vault of History to Jailbreak Large Language Modelscited by 0
  27. 2025Doppelganger Method: Breaking Role Consistency in LLM Agent via Prompt-based Transferable Adversarial Attackcited by 0
  28. 2025Can MLLMs Detect Phishing? A Comprehensive Security Benchmark Suite Focusing on Dynamic Threats and Multimodal Evaluation in Academic Environmentscited by 0