mlsanityData to 7 Oct 2026

Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Types

arXiv 2604.09544 ↗First posted 10 Apr 2026cs.CL cs.AI cs.LGcited by 3

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Large language models remain vulnerable to jailbreaks that elicit harmful responses, yet the mechanism behind harmful response generation is poorly understood. Here, we investigate how this capability is organized within model parameters. We identify and prune parameters that specifically support harmful compliance, providing a direct mechanistic analysis at the parameter level. We find that this capability depends on a sparse set of critical parameters: pruning these parameters substantially reduces harmful compliance while causing only limited degradation in benign capabilities, suggesting that key components of harmful generation are separable from those of general utility. Parameters identified from one harm category also reduce harmful responses in others, indicating components shared across harm types. This separability appears primarily in aligned models, suggesting that alignment training internally reshapes the harmful response mechanism even when behavioral safeguards remain brittle. We further show that harmful response generation is dissociable from the ability to recognize and reason about harmfulness. Finally, we extend our analysis to emergent misalignment and identify a sparse set of parameters contributing to it, with substantial sharing across fine-tuning domains. Together, these results reveal a consistent parameter-level organization underlying unsafe behaviors and point toward more principled interventions for improving model safety.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 52 of its 75 references that are in the map. First posted on arXiv 10 Apr 2026.

Builds on

52 of 75 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025LLMs Encode Harmfulness and Refusal Separately16 Jul
  2. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Study20 May
  3. 2025Representation Bending for Large Language Model Safety2 Apr
  4. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs24 Feb
  5. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations17 Jun
  6. 2024Refusal in Language Models Is Mediated by a Single Direction17 Jun
  7. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  8. 2024Improving Alignment and Robustness with Circuit Breakers6 Jun
  9. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 42

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  9. 2022Editing Models with Task Arithmeticcited by 1,076
  10. 2019Machine Unlearningcited by 1,020
  11. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  12. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
30 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  3. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  4. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  5. 2024A StrongREJECT for Empty Jailbreakscited by 352
  6. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  7. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  8. 2023An Overview of Catastrophic AI Riskscited by 214
  9. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  10. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  11. 2018The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigationcited by 159
  12. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  13. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  14. 2025Model Organisms for Emergent Misalignmentcited by 84
  15. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 77
  16. 2025Persona Features Control Emergent Misalignmentcited by 76
  17. 2024Defending LLMs against Jailbreaking Attacks via Backtranslationcited by 65
  18. 2025Command A: An Enterprise-Ready Large Language Modelcited by 57
  19. 2024The Super Weight in Large Language Modelscited by 51
  20. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  21. 2023Spear Phishing With Large Language Modelscited by 49
  22. 2024Dissecting Language Models: Machine Unlearning via Selective Pruningcited by 38
  23. 2025Understanding Refusal in Language Models with Sparse Autoencoderscited by 26
  24. 2023Will releasing the weights of future large language models grant widespread access to pandemic agents?cited by 25
  25. 2026Emergent Misalignment is Easy, Narrow Misalignment is Hardcited by 21
  26. 2025Steering Language Models with Weight Arithmeticcited by 19
  27. 2025Position-aware Automatic Circuit Discoverycited by 14
  28. 2024Revisiting the Robust Alignment of Circuit Breakerscited by 10
  29. 2025PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Samplingcited by 6
  30. 2025AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbotcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.