mlsanityData to 7 Oct 2026

LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B

arXiv 2310.20624 ↗First posted 31 Oct 2023cs.LG cs.AIcited by 183

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

AI developers often apply safety alignment procedures to prevent the misuse of their AI systems. For example, before Meta released Llama 2-Chat - a collection of instruction fine-tuned large language models - they invested heavily in safety training, incorporating extensive red-teaming and reinforcement learning from human feedback. We explore the robustness of safety training in language models by subversively fine-tuning Llama 2-Chat. We employ quantized low-rank adaptation (LoRA) as an efficient fine-tuning method. With a budget of less than 200 and using only one GPU, we successfully undo the safety training of Llama 2-Chat models of sizes 7B, 13B, and 70B and on the Mixtral instruct model. Specifically, our fine-tuning technique significantly reduces the rate at which the model refuses to follow harmful instructions. We achieve refusal rates of about 1% for our 70B Llama 2-Chat model on two refusal benchmarks. Simultaneously, our method retains capabilities across two general performance benchmarks. We show that subversive fine-tuning is practical and effective, and hence argue that evaluating risks from fine-tuning should be a core part of risk assessments for releasing model weights. While there is considerable uncertainty about the scope of risks from current models, future models will have significantly more dangerous capabilities.

Key facts. Cited by 183 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 31 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

21 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  2. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 18

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  11. 2022Red Teaming Language Models with Language Modelscited by 983
  12. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
6 more
  1. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  2. 2023Language Models Represent Space and Timecited by 306
  3. 2023How to Catch an AI Liar: Lie Detection in Black-Box LLMs by Asking Unrelated Questionscited by 82
  4. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  5. 2021Non-Transferable Learning: A New Approach for Model Ownership Verification and Applicability Authorizationcited by 50
  6. 2023Will releasing the weights of future large language models grant widespread access to pandemic agents?cited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 61; 2025: 64; 2026: 57.

183 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  3. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  4. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  5. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  6. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  7. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  8. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  9. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  10. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  11. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  12. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
38 more
  1. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  2. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  3. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  4. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  5. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  6. 2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
  7. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  8. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  9. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  10. 2023Tied-Lora: Enhancing parameter efficiency of LoRA with weight tyingcited by 65
  11. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Modelcited by 60
  12. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  13. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 50
  14. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  15. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  16. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 42
  17. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  18. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 39
  19. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  20. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  21. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  22. 2024Immunization against harmful fine-tuning attackscited by 33
  23. 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Samplingcited by 33
  24. 2024Crabs: Consuming Resource via Auto-generation for LLM-DoS Attack under Black-box Settingscited by 33
  25. 2024Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignmentcited by 31
  26. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 31
  27. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  28. 2024NetSafe: Exploring the Topological Safety of Multi-agent Networkscited by 29
  29. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  30. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  31. 2025Representation Bending for Large Language Model Safetycited by 28
  32. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  33. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 25
  34. 2024LoRATK: LoRA Once, Backdoor Everywhere in the Share-and-Play Ecosystemcited by 22
  35. 2024From LLMs to MLLMs: Exploring the Landscape of Multimodal Jailbreakingcited by 22
  36. 2024Can Editing LLMs Inject Harm?cited by 22
  37. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  38. 2024Towards Trustworthy AI: A Review of Ethical and Robust Large Language Modelscited by 21

50 most cited of 183 All 183 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  2. 2025Noise Injection Systemically Degrades Large Language Model Safety Guardrails
  3. 2025Decoupling Safety into Orthogonal Subspace: Cost-Efficient and Performance-Preserving Alignment for Large Language Models
  4. 2024From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards
  5. 2025Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Training
  6. 2024Applying Refusal-Vector Ablation to Llama 3.1 70B Agents
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback
  8. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models