mlsanityData to 7 Oct 2026

On the Role of Attention Heads in Large Language Model Safety

arXiv 2410.13708 ↗First posted 17 Oct 2024cs.CL cs.AI cs.CRcited by 70

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large language models (LLMs) achieve state-of-the-art performance on multiple language tasks, yet their safety guardrails can be circumvented, leading to harmful generations. In light of this, recent research on safety mechanisms has emerged, revealing that when safety representations or component are suppressed, the safety capability of LLMs are compromised. However, existing research tends to overlook the safety impact of multi-head attention mechanisms, despite their crucial role in various model functionalities. Hence, in this paper, we aim to explore the connection between standard attention mechanisms and safety capability to fill this gap in the safety-related mechanistic interpretability. We propose a novel metric which tailored for multi-head attention, the Safety Head ImPortant Score (Ships), to assess the individual heads' contributions to model safety. Based on this, we generalize Ships to the dataset level and further introduce the Safety Attention Head AttRibution Algorithm (Sahara) to attribute the critical safety attention heads inside the model. Our findings show that the special attention head has a significant impact on safety. Ablating a single safety head allows aligned model (e.g., Llama-2-7b-chat) to respond to 16 times more harmful queries, while only modifying 0.006% of the parameters, in contrast to the ~ 5% modification required in previous studies. More importantly, we demonstrate that attention heads primarily function as feature extractors for safety and models fine-tuned from the same base model exhibit overlapping safety heads through comprehensive experiments. Together, our attribution approach and findings provide a novel perspective for unpacking the black box of safety mechanisms within large models.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 65 of its 81 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

65 of 81 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Course-Correction: Safety Alignment Using Synthetic Preferences23 Jul
  2. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons20 Jun
  3. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  4. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks25 May
  5. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  6. 2024On Prompt-Driven Safeguarding for Large Language Models31 Jan
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 58

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
38 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2024Qwen2 Technical Reportcited by 2,076
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  8. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  9. 2019Are Sixteen Heads Really Better than One?cited by 803
  10. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  12. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  13. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  14. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  15. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  16. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  17. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  18. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  19. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  20. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  21. 2021Causal Abstractions of Neural Networkscited by 401
  22. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  23. 2023Explainability for Large Language Models: A Surveycited by 331
  24. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  25. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  26. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  27. 2023Are aligned neural networks adversarially aligned?cited by 305
  28. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  29. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  30. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  31. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  32. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  33. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  34. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  35. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  36. 2023Tracr: Compiled Transformers as a Laboratory for Interpretabilitycited by 99
  37. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  38. 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden Statescited by 92

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 27; 2026: 42.

70 papers.

Sort:

By area:

  1. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  2. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  3. 2025Reinforced Lifelong Editing for Language Modelscited by 33
  4. 2025AGENTSAFE: Benchmarking the Safety of Embodied Agents on Hazardous Instructionscited by 33
  5. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  6. 2025Understanding and Rectifying Safety Perception Distortion in VLMscited by 17
  7. 2025ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Modelscited by 17
  8. 2025SAFEx: Analyzing Vulnerabilities of MoE-Based LLMs via Stable Safety-critical Expert Identificationcited by 14
  9. 2025SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agentscited by 14
  10. 2025Chain-of-Thought Hijackingcited by 13
  11. 2025Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025cited by 11
  12. 2025Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?cited by 11
38 more
  1. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  2. 2026Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Unitscited by 10
  3. 2026Formal Mechanistic Interpretability: Automated Circuit Discovery with Provable Guaranteescited by 10
  4. 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Modelscited by 9
  5. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 8
  6. 2025Investigating CoT Monitorability in Large Reasoning Modelscited by 8
  7. 2025Hidden in the Noise: Unveiling Backdoors in Audio LLMs Alignment through Latent Acoustic Pattern Triggerscited by 7
  8. 2026Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMscited by 6
  9. 2026Attention Smoothing Is All You Need For Unlearningcited by 6
  10. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
  11. 2025Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Modelscited by 4
  12. 2026SafeSeek: Universal Attribution of Safety Circuits in Language Modelscited by 4
  13. 2026(How) Learning Rates Regulate Catastrophic Overtrainingcited by 4
  14. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMscited by 3
  15. 2025CircuitSeer: Mining High-Quality Data by Probing Mathematical Reasoning Circuits in LLMscited by 3
  16. 2026AISA: Awakening Intrinsic Safety Awareness in Large Language Models against Jailbreak Attackscited by 3
  17. 2026Attention in Space: Functional Roles of VLM Heads for Spatial Reasoningcited by 3
  18. 2026SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignmentcited by 3
  19. 2025ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Modelscited by 2
  20. 2025Focusing on Language: Revealing and Exploiting Language Attention Heads in Multilingual Large Language Modelscited by 2
  21. 2025WebRec: Enhancing LLM-based Recommendations with Attention-guided RAG from Webcited by 2
  22. 2025Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modulescited by 2
  23. 2026Risk Awareness Injection: Calibrating Vision-Language Models for Safety without Compromising Utilitycited by 2
  24. 2026Steer2Edit: From Activation Steering to Component-Level Editingcited by 2
  25. 2026Steering and Rectifying Latent Representation Manifolds in Frozen Multi-modal LLMs for Video Anomaly Detectioncited by 2
  26. 2026Compressed Sensing for Capability Localization in Large Language Modelscited by 2
  27. 2025ASGuard: Activation-Scaling Guard to Mitigate Targeted Jailbreaking Attackcited by 1
  28. 2026ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Managementcited by 1
  29. 2026Depth Charge: Jailbreak Large Language Models from Deep Safety Attention Headscited by 1
  30. 2026Knowing without Acting: The Disentangled Geometry of Safety Mechanisms in Large Language Modelscited by 1
  31. 2026Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMscited by 1
  32. 2026The Salami Slicing Threat: Exploiting Cumulative Risks in LLM Systemscited by 1
  33. 2026How Do LLMs and VLMs Understand Viewpoint Rotation Without Vision? An Interpretability Studycited by 1
  34. 2026Perturbation Probing: A Two-Pass-per-Prompt Diagnostic for FFN Behavioral Circuits in Aligned LLMscited by 1
  35. 2026Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Modelscited by 1
  36. 2024Cut the Deadwood Out: Backdoor Purification via Guided Module Substitutioncited by 0
  37. 2025VisuoAlign: Safety Alignment of LVLMs with Multimodal Tree Searchcited by 0
  38. 2025Sparse Neuron Ablation Triggers Catastrophic Collapse of the Language Core in Large Vision-Language Modelscited by 0

50 most cited of 70 All 70 (JSON)