mlsanityData to 7 Oct 2026

Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring

arXiv 2410.21083 ↗First posted 28 Oct 2024cs.CL cs.AIcited by 10

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language model (LLM) safety is a critical issue, with numerous studies employing red team testing to enhance model security. Among these, jailbreak methods explore potential vulnerabilities by crafting malicious prompts that induce model outputs contrary to safety alignments. Existing black-box jailbreak methods often rely on model feedback, repeatedly submitting queries with detectable malicious instructions during the attack search process. Although these approaches are effective, the attacks may be intercepted by content moderators during the search process. We propose an improved transfer attack method that guides malicious prompt construction by locally training a mirror model of the target black-box model through benign data distillation. This method offers enhanced stealth, as it does not involve submitting identifiable malicious instructions to the target model during the search phase. Our approach achieved a maximum attack success rate of 92%, or a balanced value of 80% with an average of 1.5 detectable jailbreak queries per sample against GPT-3.5 Turbo on a subset of AdvBench. These results underscore the need for more robust defense mechanisms.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 31 of its 51 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

31 of 51 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey5 Jul
  2. 2024RL-JACK: Reinforcement Learning-powered Black-box Jailbreaking Attack against LLMs13 Jun
  3. 2024Towards Universal and Black-Box Query-Response Only Attack on LLMs with QROA4 Jun
  4. 2024Investigating Adversarial Trigger Transfer in Large Language Models24 Apr
  5. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  7. 2024Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychology24 Feb
  8. 2024All in How You Ask for It: Simple Black-Box Method for Jailbreak Attacks18 Jan
  9. 2023JailGuard: A Universal Detection Framework for LLM Prompt-based Attacks17 Dec
  10. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  12. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
4 more
  1. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
  2. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
  3. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  4. 2023Detecting Language Model Attacks with Perplexity27 Aug

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  8. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  9. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  10. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
3 more
  1. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  2. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  3. 2023TrojLLM: A Black-box Trojan Prompt Attack on Large Language Modelscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2025AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defendercited by 37
  2. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 7
  3. 2026Structured Security Auditing and Robustness Enhancement for Untrusted Agent Skillscited by 6
  4. 2025Harmful Prompt Laundering: Jailbreaking LLMs with Abductive Styles and Symbolic Encodingcited by 5
  5. 2026When Safety Becomes a Vulnerability: Exploiting LLM Alignment Homogeneity for Transferable Blocking in RAGcited by 4
  6. 2026Intent Laundering: AI Safety Datasets Are Not What They Seemcited by 1
  7. 2026Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Headscited by 1
  8. 2026Knowing without Acting: The Disentangled Geometry of Safety Mechanisms in Large Language Modelscited by 1
  9. 2026Seed Hijacking of LLM Sampling and Quantum Random Number Defensecited by 1
  10. 2026On Understanding, Identifying, and Mitigating Vulnerabilities in Agentic Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Model-Editing-Based Jailbreak against Safety-aligned Large Language Models
  2. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs
  3. 2025Prefill-level Jailbreak: A Black-Box Risk Analysis of Large Language Models
  4. 2026Exploring and Developing a Pre-Model Safeguard with Draft Models
  5. 2026JailMeter: An Evidence-Based Evaluation Framework for Jailbreak Attacks on Large Language Models
  6. 2025Jailbreak Mimicry: Automated Discovery of Narrative-Based Jailbreaks for Large Language Models
  7. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing
  8. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning