mlsanityData to 7 Oct 2026

No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks

arXiv 2405.16229 ↗First posted 25 May 2024cs.CL cs.CRcited by 28

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The existing safety alignment of Large Language Models (LLMs) is found fragile and could be easily attacked through different strategies, such as through fine-tuning on a few harmful examples or manipulating the prefix of the generation results. However, the attack mechanisms of these strategies are still underexplored. In this paper, we ask the following question: textitwhile these approaches can all significantly compromise safety, do their attack mechanisms exhibit strong similarities? To answer this question, we break down the safeguarding process of an LLM when encountered with harmful instructions into three stages: (1) recognizing harmful instructions, (2) generating an initial refusing tone, and (3) completing the refusal response. Accordingly, we investigate whether and how different attack strategies could influence each stage of this safeguarding process. We utilize techniques such as logit lens and activation patching to identify model components that drive specific behavior, and we apply cross-model probing to examine representation shifts after an attack. In particular, we analyze the two most representative types of attack approaches: Explicit Harmful Attack (EHA) and Identity-Shifting Attack (ISA). Surprisingly, we find that their attack mechanisms diverge dramatically. Unlike ISA, EHA tends to aggressively target the harmful recognition stage. While both EHA and ISA disrupt the latter two stages, the extent and mechanisms of their attacks differ significantly. Our findings underscore the importance of understanding LLMs' internal safeguarding process and suggest that diverse defense mechanisms are required to effectively cope with various types of attacks.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 64 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

59 of 64 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusal25 Apr
  2. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  3. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  4. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  5. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  8. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  9. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  10. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Models1 Sep
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  8. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
35 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2023Steering Language Models With Activation Engineeringcited by 824
  3. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  4. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  6. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  7. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  8. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  9. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  10. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  11. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  12. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  13. 2023Language Models Represent Space and Timecited by 306
  14. 2023Pretraining Language Models with Human Preferencescited by 245
  15. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  16. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  17. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  18. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  19. 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 140
  20. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  21. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  22. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  23. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  24. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  25. 2023Successor Heads: Recurring, Interpretable Attention Heads In The Wildcited by 77
  26. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  27. 2024EasyJailbreak: A Unified Framework for Jailbreaking Large Language Modelscited by 57
  28. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  29. 2023Self-Detoxifying Language Models via Toxification Reversalcited by 45
  30. 2024Immunization against harmful fine-tuning attackscited by 33
  31. 2024ROSE Doesn't Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decodingcited by 32
  32. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  33. 2024PARDEN, Can You Repeat That? Defending against Jailbreaks via Repetitioncited by 29
  34. 2023Language Model Unalignment: Parametric Red-Teaming to Expose Hidden Harms and Biasescited by 21
  35. 2023Making Harmful Behaviors Unlearnable for Large Language Modelscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 13; 2026: 5.

28 papers.

Sort:

By area:

  1. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  2. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  3. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  4. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  5. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  6. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  7. 2025AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defendercited by 37
  8. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  9. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  10. 2024Locking Down the Finetuned LLMs Safetycited by 22
  11. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  12. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
16 more
  1. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  2. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  3. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  4. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  5. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  6. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  7. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instancecited by 8
  8. 2025SPIRIT: Patching Speech Language Models against Jailbreak Attackscited by 7
  9. 2026Understanding and Preserving Safety in Fine-Tuned LLMscited by 7
  10. 2026Steered LLM Activations are Non-Surjectivecited by 7
  11. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
  12. 2024PEFT-as-an-Attack! Jailbreaking Language Models during Federated Parameter-Efficient Fine-Tuningcited by 5
  13. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Regioncited by 5
  14. 2025Matching Ranks Over Probability Yields Truly Deep Safety Alignmentcited by 0
  15. 2026Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuningcited by 0
  16. 2026Behind Harmful Compliance: Behavioral and Mechanistic Divergence Across LLM Jailbreakscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directions
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep
  3. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attacks
  4. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  5. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  6. 2025Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs
  7. 2025Decoding One Safety Trigger Token for Balancing Safety and Usability in Large Language Models
  8. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts