mlsanityData to 14 Oct 2026

Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons

arXiv 2406.14144 ↗First posted 20 Jun 2024cs.CL cs.AI cs.LGcited by 53

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 2 more
Abstract (arXiv)

Large language models (LLMs) excel in various capabilities but pose safety risks such as generating harmful content and misinformation, even after safety alignment. In this paper, we explore the inner mechanisms of safety alignment through the lens of mechanistic interpretability, focusing on identifying and analyzing safety neurons within LLMs that are responsible for safety behaviors. We propose inference-time activation contrasting to locate these neurons and dynamic activation patching to evaluate their causal effects on model safety. Experiments on multiple prevalent LLMs demonstrate that we can consistently identify about 5% safety neurons, and by only patching their activations we can restore over 90% of the safety performance across various red-teaming benchmarks without influencing general ability. The finding of safety neurons also helps explain the ''alignment tax'' phenomenon by revealing that the key neurons for model safety and helpfulness significantly overlap, yet they require different activation patterns for the same neurons. Furthermore, we demonstrate an application of our findings in safeguarding LLMs by detecting unsafe outputs before generation. The source code is available at https://github.com/THU-KEG/SafetyNeuron.

Key facts. Cited by 53 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 71 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

50 of 71 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  2. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity3 Jan
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  4. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  12. 2024Mixtral of Expertscited by 1,823
32 more
  1. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  4. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  5. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  9. 2022Toy Models of Superpositioncited by 971
  10. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  12. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  13. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  14. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  15. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  16. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  17. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  18. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  19. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  20. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  21. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  22. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  23. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
  24. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  25. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 116
  26. 2024Universal Neurons in GPT2 Language Modelscited by 102
  27. 2023Aligning Large Language Models with Human Preferences through Representation Engineeringcited by 71
  28. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 66
  29. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 64
  30. 2024VB-LoRA: Extreme Parameter Efficient Fine-Tuning with Vector Bankscited by 52
  31. 2024ADELIE: Aligning Large Language Models on Information Extractioncited by 24
  32. 2023Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHFcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 14; 2026: 34.

53 papers.

Sort:

By area:

  1. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  2. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  3. 2024JailbreakLens: Interpreting Jailbreak Mechanism in the Lens of Representation and Circuitcited by 22
  4. 2025TwinBreak: Jailbreaking LLM Security Alignments based on Twin Promptscited by 19
  5. 2025ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Modelscited by 17
  6. 2025Advancing LLM Safe Alignment with Safety Representation Rankingcited by 16
  7. 2025Chain-of-Thought Hijackingcited by 16
  8. 2024Superficial Safety Alignment Hypothesiscited by 14
  9. 2025Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoningcited by 11
  10. 2025Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metriccited by 11
  11. 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Modelscited by 10
  12. 2025Backdoor Attribution: Elucidating and Controlling Backdoor in Language Modelscited by 10
38 more
  1. 2026Mechanistic Data Attribution: Tracing the Training Origins of Interpretable LLM Unitscited by 10
  2. 2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contextscited by 9
  3. 2024Understanding the Effectiveness of Coverage Criteria for Large Language Models: A Special Angle from Jailbreak Attackscited by 8
  4. 2025Safety Instincts: LLMs Learn to Trust Their Internal Compass for Self-Defensecited by 4
  5. 2025Efficiency vs. Alignment: Investigating Safety and Fairness Risks in Parameter-Efficient Fine-Tuning of LLMscited by 4
  6. 2026Large Language Lobotomy: Jailbreaking Mixture-of-Experts via Expert Silencingcited by 4
  7. 2026SafeSeek: Universal Attribution of Safety Circuits in Language Modelscited by 4
  8. 2026A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Modelscited by 3
  9. 2026Fail-Closed Alignment for Large Language Modelscited by 2
  10. 2026Alignment-Weighted DPO: A principled reasoning approach to improve safety alignmentcited by 2
  11. 2026Compressed Sensing for Capability Localization in Large Language Modelscited by 2
  12. 2026Do All LLMs Know When They're Being Harmful? A Reproducibility Study of Latent-Space Safety Probes Across Model Familiescited by 2
  13. 2024Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyondcited by 1
  14. 2025NeuroBreak: Unveil Internal Jailbreak Mechanisms in Large Language Modelscited by 1
  15. 2025Patching LLM Like Software: A Lightweight Method for Improving Safety Policy in Large Language Modelscited by 1
  16. 2026RAPO: Risk-Aware Preference Optimization for Generalizable Safe Reasoningcited by 1
  17. 2026Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMscited by 1
  18. 2026Prune, Interpret, Evaluate: A Cross-Layer Transcoder-Native Framework for Efficient Circuit Discovery via Feature Attributioncited by 1
  19. 2026Hidden in the Request: Explaining Unethical LLM Compliance through Token Relevancecited by 1
  20. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustnesscited by 1
  21. 2026Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliterationcited by 1
  22. 2026Safety Beyond the Interface: Detecting Harm via Latent States in Large Language Modelscited by 1
  23. 2025Neuron-level Balance between Stability and Plasticity in Deep Reinforcement Learningcited by 0
  24. 2026Symmetry Defeats Auditingcited by 0
  25. 2026Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Modelscited by 0
  26. 2026Understanding helpfulness and harmless tension in reward modelscited by 0
  27. 2026Leverage Is Not Reach: A Control-Window Law for Single-Neuron Steering in Language Modelscited by 0
  28. 2026ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstructioncited by 0
  29. 2026Efficient LLM Adversarial Training via Low-Rank Defense and Circuit-Guided Surrogatescited by 0
  30. 2026Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languagescited by 0
  31. 2026Why Are LLM Backdoor Defenses Fragmented? A Feature-Level Explanation with Sparse Autoencoderscited by 0
  32. 2026Do VLMs Share Safety Neurons Across Modalities?cited by 0
  33. 2026SEAL: Reinforcing Global Safety in Mixture-of-Experts through Shared Expert ALignmentcited by 0
  34. 2026First Token Matters: Understanding Safety Collapse in Large Reasoning Modelscited by 0
  35. 2026Multilingual Safety Signals Are Multi-Layered: Filtering Safety-Degrading Data for Safer LLMscited by 0
  36. 2026From Latents to Wires: Surgical Post-Editing on Large Language Modelscited by 0
  37. 2026When Do Model Internals Help? Exploring the Role of Representation Engineering in LLM Safetycited by 0
  38. 2026Tool Mediation Alters Refusal Mechanisms in Large Language Modelscited by 0

50 most cited of 53 All 53 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SafeNeuron: Neuron-Level Safety Alignment for Large Language Models
  2. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  3. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  4. 2024Locking Down the Finetuned LLMs Safety
  5. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
  6. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  7. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing
  8. 2026Who Bridges Safety? Identifying and Targeting Cross-Lingual Shared Safety Pathways