mlsanityData to 7 Oct 2026

Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications

arXiv 2402.05162 ↗First posted 7 Feb 2024cs.LG cs.AI cs.CLcited by 220

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Large language models (LLMs) show inherent brittleness in their safety mechanisms, as evidenced by their susceptibility to jailbreaking and even non-malicious fine-tuning. This study explores this brittleness of safety alignment by leveraging pruning and low-rank modifications. We develop methods to identify critical regions that are vital for safety guardrails, and that are disentangled from utility-relevant regions at both the neuron and rank levels. Surprisingly, the isolated regions we find are sparse, comprising about 3% at the parameter level and 2.5% at the rank level. Removing these regions compromises safety without significantly impacting utility, corroborating the inherent brittleness of the model's safety mechanisms. Moreover, we show that LLMs remain vulnerable to low-cost fine-tuning attacks even when modifications to the safety-critical regions are restricted. These findings underscore the urgent need for more robust safety strategies in LLMs.

Key facts. Cited by 220 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 96 of its 115 references that are in the map. First posted on arXiv 7 Feb 2024.

Builds on

96 of 115 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
  2. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity3 Jan
  3. 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
  4. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  6. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  7. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  9. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun

Other references · 87

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  9. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  12. 2017Learning Important Features Through Propagating Activation Differencescited by 1,454
  13. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  14. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  15. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  16. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  17. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  18. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  19. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  20. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  21. 2019Are Sixteen Heads Really Better than One?cited by 803
  22. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  23. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  24. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  25. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  26. 2024Self-Rewarding Language Modelscited by 654
  27. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 643
  28. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  29. 2019Designing and Interpreting Probes with Control Taskscited by 608
  30. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  31. 2016Learning Structured Sparsity in Deep Neural Networkscited by 538
  32. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  33. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  34. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  35. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  36. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  37. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  38. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 438

50 most cited of 87 All 87 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 61; 2025: 90; 2026: 69.

220 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  3. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  4. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  5. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  6. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  7. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  8. 2024On Memorization of Large Language Models in Logical Reasoningcited by 114
  9. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  10. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
  11. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  12. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
38 more
  1. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  2. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  3. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  4. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  5. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  6. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  7. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  8. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  9. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  10. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 54
  11. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 54
  12. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 51
  13. 2024On Large Language Model Continual Unlearningcited by 51
  14. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  15. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 50
  16. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 47
  17. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  18. 2024A safety realignment framework via subspace-oriented model fusion for large language modelscited by 40
  19. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
  20. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 38
  21. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
  22. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  23. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  24. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  25. 2024Immunization against harmful fine-tuning attackscited by 33
  26. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 33
  27. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  28. 2024Preference Tuning For Toxicity Mitigation Generalizes Across Languagescited by 31
  29. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  30. 2025Representation Bending for Large Language Model Safetycited by 28
  31. 2024Competition Report: Finding Universal Jailbreak Backdoors in Aligned LLMscited by 27
  32. 2025The Jailbreak Tax: How Useful are Your Jailbreak Outputs?cited by 26
  33. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  34. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  35. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  36. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  37. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuningcited by 20
  38. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 19

50 most cited of 220 All 220 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  2. 2026Understanding the Effects of Safety Unalignment on Large Language Models
  3. 2026NeuronGuard: Robust LLM Safety Alignment via Ablation-Aware Safety Signal Redistribution
  4. 2026SafeNeuron: Neuron-Level Safety Alignment for Large Language Models
  5. 2025SABER: Uncovering Vulnerabilities in Safety Alignment via Cross-Layer Residual Connection
  6. 2026SafeSeek: Universal Attribution of Safety Circuits in Language Models
  7. 2026When Safety Speaks a Language: A Mechanistic Analysis of Safety-Language Identity Entanglement in LLMs
  8. 2026SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models