mlsanityData to 7 Oct 2026

Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!

arXiv 2310.03693 ↗First posted 5 Oct 2023cs.CL cs.AI cs.CRcited by 1,106

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Optimizing large language models (LLMs) for downstream use cases often involves the customization of pre-trained LLMs through further fine-tuning. Meta's open release of Llama models and OpenAI's APIs for fine-tuning GPT-3.5 Turbo on custom datasets also encourage this practice. But, what are the safety costs associated with such custom fine-tuning? We note that while existing safety alignment infrastructures can restrict harmful behaviors of LLMs at inference time, they do not cover safety risks when fine-tuning privileges are extended to end-users. Our red teaming studies find that the safety alignment of LLMs can be compromised by fine-tuning with only a few adversarially designed training examples. For instance, we jailbreak GPT-3.5 Turbo's safety guardrails by fine-tuning it on only 10 such examples at a cost of less than 0.20 via OpenAI's APIs, making the model responsive to nearly any harmful instructions. Disconcertingly, our research also reveals that, even without malicious intent, simply fine-tuning with benign and commonly used datasets can also inadvertently degrade the safety alignment of LLMs, though to a lesser extent. These findings suggest that fine-tuning aligned LLMs introduces new safety risks that current safety infrastructures fall short of addressing -- even if a model's initial safety alignment is impeccable, it is not necessarily to be maintained after custom fine-tuning. We outline and critically analyze potential mitigations and advocate for further research efforts toward reinforcing safety protocols for the custom fine-tuning of aligned LLMs.

Key facts. Cited by 1,106 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 66 of its 92 references that are in the map. First posted on arXiv 5 Oct 2023.

Builds on

66 of 92 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  2. 2023Are aligned neural networks adversarially aligned?26 Jun
  3. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Models22 Jun
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 62

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  10. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  11. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  12. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  13. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  14. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  15. 2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,052
  16. 2022Red Teaming Language Models with Language Modelscited by 983
  17. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  18. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  19. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  20. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  21. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  22. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  23. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Studycited by 478
  24. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  25. 2020Backdoor Learning: A Surveycited by 412
  26. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  27. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  28. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  29. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  30. 2023Data Selection for Language Models via Importance Resamplingcited by 322
  31. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  32. 2019A backdoor attack against LSTM-based text classification systemscited by 214
  33. 2020BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvementscited by 191
  34. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  35. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  36. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  37. 2023Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Modelcited by 115
  38. 2023Foundation Models and Fair Usecited by 112

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 339; 2025: 382; 2026: 364.

1,106 papers.

Sort:

By area:

  1. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2024A Survey on Large Language Models for Code Generationcited by 646
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  6. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  7. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  9. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  10. 2024A StrongREJECT for Empty Jailbreakscited by 352
  11. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  12. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
38 more
  1. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  2. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  3. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  4. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  5. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  6. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  7. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  8. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  9. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  10. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  11. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  12. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  13. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  14. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  15. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  16. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  17. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  18. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 149
  19. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  20. 2024Continual Learning for Large Language Models: A Surveycited by 144
  21. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  22. 2024AutoDefense: Multi-Agent LLM Defense against Jailbreak Attackscited by 135
  23. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  24. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  25. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  26. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  27. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  28. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  29. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  30. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  31. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  32. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  33. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  34. 2024PLeak: Prompt Leaking Attacks against Large Language Model Applicationscited by 113
  35. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoningcited by 113
  36. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  37. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimizationcited by 112
  38. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agentscited by 112

50 most cited of 1,106 All 1,106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Secure LLM Fine-Tuning via Safety-Aware Probing
  2. 2024Overriding Safety protections of Open-source Models
  3. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  4. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
  5. 2024Large Language Model Safety: A Holistic Survey
  6. 2025LLM-Safety Evaluations Lack Robustness
  7. 2026Safety Is Not Universal: The Selective Safety Trap in LLM Alignment
  8. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts