mlsanityData to 7 Oct 2026

Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Heads

arXiv 2603.05772 ↗First posted 6 Mar 2026cs.CR cs.AIcited by 1

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Currently, open-sourced large language models (OSLLMs) have demonstrated remarkable generative performance. However, as their structure and weights are made public, they are exposed to jailbreak attacks even after alignment. Existing attacks operate primarily at shallow levels, such as the prompt or embedding level, and often fail to expose vulnerabilities rooted in deeper model components, which creates a false sense of security for successful defense. In this paper, we propose textbfunderlineSafety textbfunderlineAttention textbfunderlineHead textbfunderlineAttack (SAHA), an attention-head-level jailbreak framework that explores the vulnerability in deeper but insufficiently aligned attention heads. SAHA contains two novel designs. Firstly, we reveal that deeper attention layers introduce more vulnerability against jailbreak attacks. Based on this finding, SAHA introduces textitAblation-Impact Ranking head selection strategy to effectively locate the most vital layer for unsafe output. Secondly, we introduce a boundary-aware perturbation method, textiti.e. Layer-Wise Perturbation, to probe the generation of unsafe content with minimal perturbation to the attention. This constrained perturbation guarantees higher semantic relevance with the target intent while ensuring evasion. Extensive experiments show the superiority of our method: SAHA improves ASR by 14% over SOTA baselines, revealing the vulnerability of the attack surface on the attention head. Our code is available at https://anonymous.4open.science/r/SAHA.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 58 of its 70 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

58 of 70 references.

Closest ancestors · 27

Same direction, newest first

  1. 2025CAVGAN: Unifying Jailbreak and Defense of LLMs via Generative Adversarial Attacks on their Internal Representations8 Jul
  2. 2025M2S: Multi-turn to Single-turn jailbreak in Red Teaming for LLMs6 Mar
  3. 2025QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Language13 Feb
  4. 2025JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulation11 Feb
  5. 2025Injecting Universal Jailbreak Backdoors into LLMs in Minutes9 Feb
  6. 2025Smoothed Embeddings for Robust Language Models27 Jan
  7. 2024SequentialBreak: Large Language Models Can be Fooled by Embedding Jailbreak Prompts into Sequential Prompt Chains10 Nov
  8. 2024SQL Injection Jailbreak: A Structural Disaster of Large Language Models3 Nov
  9. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring28 Oct
  10. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMs3 Oct
  11. 2024Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attacks27 Aug
  12. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States9 Jun
15 more
  1. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing28 May
  2. 2024GPT-4 Jailbreaks Itself with Near-Perfect Success Using Self-Explanation21 May
  3. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
  5. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space14 Feb
  6. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspective12 Jan
  7. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  8. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily14 Nov
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  12. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  13. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  14. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  15. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  8. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  10. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  12. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
19 more
  1. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  2. 2019Revealing the Dark Secrets of BERTcited by 300
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  4. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  5. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  6. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  7. 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 63
  8. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  9. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injectioncited by 53
  10. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  11. 2025Can Indirect Prompt Injection Attacks Be Detected and Removed?cited by 43
  12. 2024Multi-property Steering of Large Language Models with Dynamic Activation Compositioncited by 35
  13. 2025Controlling Large Language Models Through Concept Activation Vectorscited by 27
  14. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 15
  15. 2025Internal Value Alignment in Large Language Models through Controlled Value Vector Activationcited by 14
  16. 2024Efficient Universal Goal Hijacking with Semantics-guided Prompt Organizationcited by 13
  17. 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoderscited by 12
  18. 2025Defense against Prompt Injection Attacks via Mixture of Encodingscited by 10
  19. 2024Adversarial Style Augmentation via Large Language Model for Robust Fake News Detectioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Crafting Reversible SFT Behaviors in Large Language Modelscited by 0