Safety Anchor: Defending Harmful Fine-tuning via Geometric Bottlenecks
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
also in 1 more
Abstract (arXiv)
The safety alignment of Large Language Models (LLMs) remains vulnerable to Harmful Fine-tuning (HFT). While existing defenses impose constraints on parameters, gradients, or internal representations, we observe that they can be effectively circumvented under persistent HFT. Our analysis traces this failure to the inherent redundancy of the high-dimensional parameter space: attackers exploit optimization trajectories that are orthogonal to defense constraints to restore harmful capabilities while deceptively adhering to safety restrictions. To address this, we propose Safety Bottleneck Regularization (SBR). SBR shifts the defensive focus from the redundant parameter space to the unembedding layer, which serves as a geometric bottleneck. By anchoring the final hidden states of harmful queries to those of the safety-aligned model, SBR enables the model to maintain safe responses even under persistent HFT. Extensive experiments confirm SBR's effectiveness, demonstrating that utilizing just a single safety anchor is sufficient to reduce the Harmful Score to <10 while preserving competitive performance on benign downstream tasks.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 50 references that are in the map. First posted on arXiv 7 May 2026.
Builds on
43 of 50 references.
Closest ancestors · 12
Same direction, newest first
- 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning17 Dec
- 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbation13 Oct
- 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning18 Aug
- 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
- 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attack28 May
- 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attack2 Feb
- 2023Universal Jailbreak Backdoors from Poisoned Human Feedback24 Nov
- 2023Removing RLHF Protections in GPT-4 via Fine-Tuning9 Nov
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
- 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
Other references · 31
- 2017Attention Is All You Needcited by 59,932
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2021Training Verifiers to Solve Math Word Problemscited by 9,743
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2022Editing Models with Task Arithmeticcited by 1,076
- 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2022Toy Models of Superpositioncited by 928
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
19 more
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
- 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
- 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
- 2023Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Timecited by 247
- 2023A mathematical perspective on Transformerscited by 160
- 2025Qwen2.5-1M Technical Reportcited by 158
- 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
- 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
- 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
- 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
- 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
- 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
- 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 48
- 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
- 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
- 2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
- 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
- 2026DF-LoGiT: Data-Free Logic-Gated Backdoor Attacks in Vision Transformerscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Merging
- 2026Understanding and Preserving Safety in Fine-Tuned LLMs
- 2025Secure LLM Fine-Tuning via Safety-Aware Probing
- 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
- 2026Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment
- 2025Gradient Surgery for Safe LLM Fine-Tuning
- 2024Safety Layers in Aligned Large Language Models: The Key to LLM Security
- 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance