Data to 7 Oct 2026

Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Models

arXiv 2405.16833 ↗First posted 27 May 2024cs.LGcited by 122

Not on the map. Why?

Cited by 68 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

While large language models (LLMs) such as Llama-2 or GPT-4 have shown impressive zero-shot performance, fine-tuning is still necessary to enhance their performance for customized datasets, domain-specific tasks, or other private needs. However, fine-tuning all parameters of LLMs requires significant hardware resources, which can be impractical for typical users. Therefore, parameter-efficient fine-tuning such as LoRA have emerged, allowing users to fine-tune LLMs without the need for considerable computing resources, with little performance degradation compared to fine-tuning all parameters. Unfortunately, recent studies indicate that fine-tuning can increase the risk to the safety of LLMs, even when data does not contain malicious content. To address this challenge, we propose Safe LoRA, a simple one-liner patch to the original LoRA implementation by introducing the projection of LoRA weights from selected layers to the safety-aligned subspace, effectively reducing the safety risks in LLM fine-tuning while maintaining utility. It is worth noting that Safe LoRA is a training-free and data-free approach, as it only requires the knowledge of the weights from the base and aligned LLMs. Our extensive experiments demonstrate that when fine-tuning on purely malicious data, Safe LoRA retains similar safety performance as the original aligned model. Moreover, when the fine-tuning dataset contains a mixture of both benign and malicious data, Safe LoRA mitigates the negative effect made by malicious data while preserving performance on downstream tasks. Our codes are available at urlhttps://github.com/IBM/SafeLoRA.

Key facts. Cited by 122 papers in the map (data through 7 Oct 2026). Builds on 51 of its 57 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

51 of 57 references.

References · 51

2022Training language models to follow instructions with human feedbackcited by 15,712
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
2017Deep reinforcement learning from human preferencescited by 3,846
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
2019Fine-Tuning Language Models from Human Preferencescited by 2,124
2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565

38 more

2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
2022Editing Models with Task Arithmeticcited by 1,076
2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
2023Steering Language Models With Activation Engineeringcited by 824
2021Robust fine-tuning of zero-shot modelscited by 785
2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
2024Self-Rewarding Language Modelscited by 654
2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
2021Merging Models with Fisher-Weighted Averagingcited by 613
2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
2022Improving alignment of dialogue agents via targeted human judgementscited by 527
2021Fast Model Editing at Scalecited by 491
2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
2022Memory-Based Model Editing at Scalecited by 443
2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
2023LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 317
2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
2023Low-Resource Languages Jailbreak GPT-4cited by 290
2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 217
2021Training Neural Networks with Fixed Sparse Maskscited by 212
2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
2022Patching open-vocabulary models by interpolating weightscited by 187
2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
2022Fusing finetuned models for better pretrainingcited by 110
2021Editing a classifier by rewriting its prediction rulescited by 64
2022In conversation with Artificial Intelligence: aligning language models with human valuescited by 61
2022Fixing Model Bugs with Natural Language Patchescited by 32
2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 19; 2025: 61; 2026: 42.

122 papers.

Sort:

By area:

2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
2024Programming Refusal with Conditional Activation Steeringcited by 191
2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
2025Model Organisms for Emergent Misalignmentcited by 84
2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 47
2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39

38 more

2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 19
2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17
2025Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Modelscited by 16
2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 15
2024Superficial Safety Alignment Hypothesiscited by 14
2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learningcited by 14
2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 14
2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 13
2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
2025Safe Pruning LoRA: Robust Distance-Guided Pruning for Safety Alignment in Adaptation of LLMscited by 10
2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 10
2025Too Good to be Bad: On the Failure of LLMs to Role-Play Villainscited by 10
2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
2026Obscure but Effective: Classical Chinese Jailbreak Prompt Optimization via Bio-Inspired Searchcited by 10
2024Decoupled Alignment for Robust Plug-and-Play Adaptationcited by 9
2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
2024Toward Secure Tuning: Mitigating Security Risks from Instruction Fine-Tuningcited by 8
2025Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuningcited by 8
2025Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignmentcited by 8
2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
2025Don't Run with Scissors: Pruning Breaks VLA Models but They Can Be Recoveredcited by 8
2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
2025Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraintcited by 6

50 most cited of 122 All 122 (JSON)