mlsanityData to 14 Oct 2026

Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraint

arXiv 2509.06795 ↗First posted 8 Sep 2025cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Instruction Fine-Tuning (IFT) has been widely adopted as an effective post-training strategy to enhance various abilities of Large Language Models (LLMs). However, prior studies have shown that IFT can significantly compromise LLMs' safety, particularly their ability to refuse malicious instructions, raising significant concerns. Recent research into the internal mechanisms of LLMs has identified the refusal direction (r-direction) in the hidden states, which plays a pivotal role in governing refusal behavior. Building on this insight, our study reveals that the r-direction tends to drift during training, which we identify as one of the causes of the associated safety risks. To mitigate such drift, our proposed ProCon method introduces a projection-constrained loss term that regularizes the projection magnitude of each training sample's hidden state onto the r-direction. Our initial analysis shows that applying an appropriate constraint can effectively mitigate the refusal direction drift and associated safety risks, but remains limited by overall performance barriers. To overcome this barrier, informed by our observation of early-stage sharp drift and a data-driven perspective, we introduce a warm-up strategy that emphasizes early-stage strong constraints and broaden the data distribution to strengthen constraint signals, leading to an enhanced ProCon method. Experimental results under various datasets, scenarios, and LLMs demonstrate that our method can significantly mitigate safety risks posed by IFT while preserving task performance gains. Even compared with strong baselines, our method consistently delivers superior overall performance. Crucially, our analysis indicates that ProCon can contribute to stabilizing the r-direction during training, while such an interpretability-driven exploration of LLMs' internal mechanisms lays a solid foundation for future safety research.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 45 of its 51 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

45 of 51 references.

References · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  11. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
33 more
  1. 2022Red Teaming Language Models with Language Modelscited by 1,026
  2. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  4. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  5. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 694
  6. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  7. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  8. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  9. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  10. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 192
  11. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  12. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  13. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  14. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  15. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  16. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  17. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 114
  18. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmeticcited by 112
  19. 2024A Comprehensive Study of Jailbreak Attack versus Defense for Large Language Modelscited by 106
  20. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  21. 2018Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approachcited by 91
  22. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  23. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 89
  24. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 76
  25. 2025Unlocking Efficient Long-to-Short LLM Reasoning with Model Mergingcited by 58
  26. 2019Where is the Information in a Deep Neural Network?cited by 42
  27. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  28. 2024SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuningcited by 32
  29. 2025Linear Representations of Political Perspective Emerge in Large Language Modelscited by 32
  30. 2024Foot In The Door: Understanding Large Language Model Jailbreaking via Cognitive Psychologycited by 27
  31. 2023Analyzing the Inherent Response Tendency of LLMs: Real-World Instructions-Driven Jailbreakcited by 23
  32. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 20
  33. 2025Directional Gradient Projection for Robust Fine-Tuning of Foundation Modelscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2025Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Educationcited by 5
  2. 2026Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Stalenesscited by 5
  3. 2026S3-CoT: Self-Sampled Succinct Reasoning Enables Efficient Chain-of-Thought LLMscited by 4
  4. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  5. 2026Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometrycited by 2
  6. 2026From Refusal Geometry to Safety Geometry: Harmfulness--Refusal Coupling under Dynamic Adversarial Fine-Tuningcited by 0
  7. 2026How Narrative Wrapping Affects LLM Refusal: A Cross-Language Benchmark and Defensecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Training
  2. 2026When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning
  3. 2026RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMs
  4. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attacks
  5. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
  6. 2026Decomposing Refusal Steering in Mixture-of-Experts Models
  7. 2026The Geometry of Refusal: Linear Instability in Safety-Aligned LLMs
  8. 2025Differentiated Directional Intervention A Framework for Evading LLM Safety Alignment