mlsanityData to 14 Oct 2026

BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B

arXiv 2311.00117 ↗First posted 31 Oct 2023cs.CLcited by 36

Not on the map. Why?

Abstract (arXiv)

Llama 2-Chat is a collection of large language models that Meta developed and released to the public. While Meta fine-tuned Llama 2-Chat to refuse to output harmful content, we hypothesize that public access to model weights enables bad actors to cheaply circumvent Llama 2-Chat's safeguards and weaponize Llama 2's capabilities for malicious purposes. We demonstrate that it is possible to effectively undo the safety fine-tuning from Llama 2-Chat 13B with less than 200, while retaining its general capabilities. Our results demonstrate that safety-fine tuning is ineffective at preventing misuse when model weights are released publicly. Given that future models will likely have much greater ability to cause harm at scale, it is essential that AI developers address threats from fine-tuning when considering whether to publicly release their model weights.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 13 of its 15 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

13 of 15 references.

References · 13

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  8. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  9. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  10. 2023Eight Things to Know about Large Language Modelscited by 73
  11. 2023Can large language models democratize access to dual-use biotechnology?cited by 35
  12. 2022Targeted Phishing Campaigns using Large Scale Language Modelscited by 23
1 more
  1. 2023Devising and Detecting Phishing: Large Language Models vs. Smaller Human Modelscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 19; 2025: 13; 2026: 3.

36 papers.

Sort:

By area:

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  2. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  3. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  4. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  5. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  6. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  7. 2024Large Language Model Safety: A Holistic Surveycited by 59
  8. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  9. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 45
  10. 2024Immunization against harmful fine-tuning attackscited by 35
  11. 2025DarkBench: Benchmarking Dark Patterns in Large Language Modelscited by 28
  12. 2025RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?cited by 24
24 more
  1. 2024Locking Down the Finetuned LLMs Safetycited by 22
  2. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  3. 2025SDD: Self-Degraded Defense against Malicious Fine-tuningcited by 13
  4. 2024A Chinese Dataset for Evaluating the Safeguards in Large Language Modelscited by 12
  5. 2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 11
  6. 2024Societal Adaptation to Advanced AIcited by 10
  7. 2024The Necessity of AI Audit Standards Boardscited by 9
  8. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 9
  9. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  10. 2023Hazards from Increasingly Accessible Fine-Tuning of Downloadable Foundation Modelscited by 7
  11. 2024SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesiscited by 7
  12. 2024Toxicity of the Commons: Curating Open-Source Pre-Training Datacited by 7
  13. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 6
  14. 2024Applying Refusal-Vector Ablation to Llama 3.1 70B Agentscited by 6
  15. 2025SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoningcited by 4
  16. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  17. 2025AI Companies Should Report Pre- and Post-Mitigation Safety Evaluationscited by 3
  18. 2024Risk Sources and Risk Management Measures in Support of Standards for General-Purpose AI Systemscited by 2
  19. 2025The Pitfalls of "Security by Obscurity" And What They Mean for Transparent AIcited by 2
  20. 2025A Red Teaming Roadmap Towards System-Level Safetycited by 2
  21. 2026DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuningcited by 1
  22. 2025From Threat to Tool: Leveraging Refusal-Aware Injection Attacks for Safety Alignmentcited by 0
  23. 2025Rethinking Deep Alignment Through The Lens Of Incomplete Learningcited by 0
  24. 2026Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Textcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
  2. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions
  3. 2024From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
  5. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
  6. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  7. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation
  8. 2026Self-Mined Hardness for Safety Fine-Tuning