mlsanityData to 14 Oct 2026

Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions

arXiv 2309.07875 ↗First posted 14 Sep 2023cs.CLcited by 387

Not on the map. Why?

Cited by 206 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Training large language models to follow instructions makes them perform better on a wide range of tasks and generally become more helpful. However, a perfectly helpful model will follow even the most malicious instructions and readily generate harmful content. In this paper, we raise concerns over the safety of models that only emphasize helpfulness, not harmlessness, in their instruction-tuning. We show that several popular instruction-tuned models are highly unsafe. Moreover, we show that adding just 3% safety examples (a few hundred demonstrations) when fine-tuning a model like LLaMA can substantially improve its safety. Our safety-tuning does not make models significantly less capable or helpful as measured by standard benchmarks. However, we do find exaggerated safety behaviours, where too much safety-tuning makes models refuse perfectly safe prompts if they superficially resemble unsafe ones. As a whole, our results illustrate trade-offs in training LLMs to be helpful and training them to be safe.

Key facts. Cited by 387 papers in the map (data through 14 Oct 2026). Builds on 35 of its 47 references that are in the map. First posted on arXiv 14 Sep 2023.

Builds on

35 of 47 references.

References · 35

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
23 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  11. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  12. 2023LIMA: Less Is More for Alignmentcited by 1,100
  13. 2021Ethical and social risks of harm from Language Modelscited by 966
  14. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  15. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  16. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  17. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  18. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  19. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 360
  20. 2023The False Promise of Imitating Proprietary LLMscited by 263
  21. 2021Human-in-the-Loop for Data Collection: a Multi-Target Counter Narrative Dataset to Fight Online Hate Speechcited by 82
  22. 2023TrustGPT: A Benchmark for Trustworthy and Responsible Large Language Modelscited by 58
  23. 2022SafeText: A Benchmark for Exploring Physical Safety in Language Modelscited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 126; 2025: 147; 2026: 105.

387 papers.

Sort:

By area:

  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  3. 2024Phi-4 Technical Reportcited by 711
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  6. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 288
  7. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 274
  8. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  9. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 244
  10. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  11. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  12. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
38 more
  1. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 167
  2. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  3. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
  4. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 130
  5. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  6. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  7. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  8. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  9. 2024Rule Based Rewards for Language Model Safetycited by 108
  10. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 107
  11. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 105
  12. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 96
  13. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  14. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 92
  15. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  16. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 90
  17. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  18. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidancecited by 83
  19. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  20. 2024Safeguarding Large Language Models: A Surveycited by 78
  21. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 77
  22. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 72
  23. 2024The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harmcited by 70
  24. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 69
  25. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  26. 2024Play Guessing Game with LLM: Indirect Jailbreak Attack with Implicit Cluescited by 68
  27. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 68
  28. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 67
  29. 2024Fight Back Against Jailbreaking via Prompt Adversarial Tuningcited by 67
  30. 2023TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Modelscited by 66
  31. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  32. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 62
  33. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  34. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 58
  35. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  36. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 52
  37. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  38. 2025Evaluating Gemini Robotics Policies in a Veo World Simulatorcited by 51

50 most cited of 387 All 387 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguards
  2. 2024Mitigating Exaggerated Safety in Large Language Models
  3. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B
  4. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistency
  5. 2025SafeSwitch: Steering Unsafe LLM Behavior via Internal Activation Signals
  6. 2023BadLlama: cheaply removing safety fine-tuning from Llama 2-Chat 13B
  7. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts
  8. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion