mlsanityData to 7 Oct 2026

Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates

arXiv 2402.18540 ↗First posted 28 Feb 2024cs.LG cs.AI cs.CLcited by 94

Not on the map. Why?

Cited by 50 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Public LLMs such as the Llama 2-Chat underwent alignment training and were considered safe. Recently Qi et al. [2024] reported that even benign fine-tuning on seemingly safe datasets can give rise to unsafe behaviors in the models. The current paper is about methods and best practices to mitigate such loss of alignment. We focus on the setting where a public model is fine-tuned before serving users for specific usage, where the model should improve on the downstream task while maintaining alignment. Through extensive experiments on several chat models (Meta's Llama 2-Chat, Mistral AI's Mistral 7B Instruct v0.2, and OpenAI's GPT-3.5 Turbo), this paper uncovers that the prompt templates used during fine-tuning and inference play a crucial role in preserving safety alignment, and proposes the ``Pure Tuning, Safe Testing'' (PTST) strategy -- fine-tune models without a safety prompt, but include it at test time. This seemingly counterintuitive strategy incorporates an intended distribution shift to encourage alignment preservation. Fine-tuning experiments on GSM8K, ChatDoctor, and OpenOrca show that PTST significantly reduces the rise of unsafe behaviors.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Builds on 44 of its 65 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

44 of 65 references.

References · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Mistral 7Bcited by 3,065
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
32 more
  1. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  3. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  5. 2023Instruction Tuning with GPT-4cited by 676
  6. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  7. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  8. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  9. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  10. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  11. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 401
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  13. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  14. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  15. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 324
  16. 2023Low-Resource Languages Jailbreak GPT-4cited by 290
  17. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  18. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  19. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  20. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  21. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  22. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 171
  23. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  24. 2024Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Modelscited by 160
  25. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritizationcited by 156
  26. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  27. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  28. 2023Fine-tuning can cripple your foundation model; preserving features may be the solutioncited by 92
  29. 2024Immunization against harmful fine-tuning attackscited by 33
  30. 2023Exploiting Novel GPT-4 APIscited by 27
  31. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 26
  32. 2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 38; 2026: 33.

94 papers.

Sort:

By area:

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  2. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  4. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  5. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 114
  6. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  7. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  8. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  9. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  10. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  11. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  12. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
38 more
  1. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  2. 2024Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generationcited by 44
  3. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  4. 2025Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignmentcited by 32
  5. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attackscited by 28
  6. 2025Representation Bending for Large Language Model Safetycited by 28
  7. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  8. 2025Fine-Tuning Lowers Safety and Disrupts Evaluation Consistencycited by 25
  9. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  10. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  11. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  12. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuningcited by 19
  13. 2025Steering Language Models with Weight Arithmeticcited by 19
  14. 2025Vulnerability-Aware Alignment: Mitigating Uneven Forgetting in Harmful Fine-Tuningcited by 17
  15. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  16. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  17. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
  18. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  19. 2025Recontextualization Mitigates Specification Gaming without Modifying the Specificationcited by 11
  20. 2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
  21. 2025Self-Destructive Language Modelcited by 10
  22. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
  23. 2025Unintended Misalignment from Agentic Fine-Tuning: Risks and Mitigationcited by 9
  24. 2024LlamaDuo: LLMOps Pipeline for Seamless Migration from Service LLMs to Small-Scale Local LLMscited by 8
  25. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  26. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
  27. 2025LookAhead Tuning: Safer Language Models via Partial Answer Previewscited by 7
  28. 2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
  29. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  30. 2025LoRA is All You Need for Safety Alignment of Reasoning LLMscited by 6
  31. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  32. 2026Did You Forget What I Asked? Prospective Memory Failures in Large Language Modelscited by 6
  33. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
  34. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Regioncited by 5
  35. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  36. 2025SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requestscited by 5
  37. 2024Proof-of-Learning with Incentive Securitycited by 4
  38. 2025Reshaping Representation Space to Balance the Safety and Over-rejection in Large Audio Language Modelscited by 4

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training
  2. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  3. 2024Locking Down the Finetuned LLMs Safety
  4. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  5. 2025Alleviating the Fear of Losing Alignment in LLM Fine-tuning
  6. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  7. 2026Understanding and Preserving Safety in Fine-Tuned LLMs
  8. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization