mlsanityData to 7 Oct 2026

Safety-Aware Fine-Tuning of Large Language Models

arXiv 2410.10014 ↗First posted 13 Oct 2024cs.CL cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Fine-tuning Large Language Models (LLMs) has emerged as a common practice for tailoring models to individual needs and preferences. The choice of datasets for fine-tuning can be diverse, introducing safety concerns regarding the potential inclusion of harmful data samples. Manually filtering or avoiding such samples, however, can be labor-intensive and subjective. To address these difficulties, we propose a novel Safety-Aware Fine-Tuning (SAFT) framework designed to automatically detect and remove potentially harmful data, by leveraging a scoring function that exploits the subspace information of harmful and benign samples. Experimental results demonstrate the efficacy of SAFT across different LLMs and varying contamination rates, achieving reductions in harmfulness of up to 27.8%. Going beyond, we delve into the mechanism of our approach and validate its versatility in addressing practical challenges in real-world scenarios.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

38 of 47 references.

References · 38

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023Mistral 7Bcited by 3,065
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
26 more
  1. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  4. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  6. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  7. 2023Instruction Tuning with GPT-4cited by 676
  8. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  9. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  10. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  11. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  12. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  13. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  14. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  15. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  16. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  17. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  18. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  19. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  20. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  21. 2023Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuningcited by 56
  22. 2024Do LLMs Know about Hallucination? An Empirical Investigation of LLM's Hidden Statescited by 50
  23. 2024Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimensioncited by 45
  24. 2024Rethinking Data Selection for Supervised Fine-Tuningcited by 35
  25. 2024PICLe: Eliciting Diverse Behaviors from Large Language Models with Persona In-Context Learningcited by 28
  26. 2024VLMGuard: Bootstrapping Malicious Prompt Detectors from Unlabeled Vision-Language Prompts in the Wildcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 24; 2026: 8.

33 papers.

Sort:

By area:

  1. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  2. 2025Qwen3Guard Technical Reportcited by 125
  3. 2024Process Reward Model with Q-Value Rankingscited by 78
  4. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  5. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  6. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  7. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  8. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  9. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
  10. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 13
  11. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  12. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
21 more
  1. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
  2. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  3. 2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 10
  4. 2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8
  5. 2025Tiered Agentic Oversight: A Hierarchical Multi-Agent System for Healthcare Safetycited by 7
  6. 2025The Geometry of Harmfulness in LLMs through Subconcept Probingcited by 7
  7. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  8. 2025LoRA is All You Need for Safety Alignment of Reasoning LLMscited by 6
  9. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projectioncited by 6
  10. 2026Sparse Models, Sparse Safety: Unsafe Routes in Mixture-of-Experts LLMscited by 6
  11. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  12. 2026Narrow Fine-Tuning Erodes Safety Alignment in Vision-Language Agentscited by 4
  13. 2025Detecting Adversarial Fine-tuning with Auditing Agentscited by 3
  14. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  15. 2025Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generationcited by 2
  16. 2026No More, No Less: Least-Privilege Language Modelscited by 1
  17. 2025Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Datacited by 0
  18. 2025Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Modelscited by 0
  19. 2026GradShield: Alignment Preserving Finetuningcited by 0
  20. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflectioncited by 0
  21. 2026PermitGPT: A Unified Generative-AI Pipeline for Construction Hazard Forecasting, Permit Prediction, and Community Impactcited by 0