mlsanityData to 7 Oct 2026

Understanding the Role of Prompt Template in Knowledge Distillation for Safety Alignment

arXiv 2609.30802 ↗First posted 25 Sep 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Prior research has demonstrated that the choice of prompt template during Supervised Fine-Tuning (SFT) significantly impacts the robustness of safety alignment afterwards. However, the influence of template selection during Knowledge Distillation (KD) from teacher to student remains largely unexplored. Thus, we fill this gap by analyzing how different template configurations influence the pre-existing safety alignment of the student. We observe a significant degradation of safety alignment present in the aligned base instruct-tuned model. Specifically, we find that utilizing chat templates renders the model more compliant with harmful queries compared to a non-chat template. These findings are consistent across three models: LLaMA, Gemma and Qwen model families and are evaluated across multiple safety benchmarks. We further show that using a non-chat template during distillation better preserves the base student's internal representations, while chat template distillation induces a larger representational shift. Code: https://github.com/anjilab/role-of-prompt-template-in-kd

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 24 references that are in the map. First posted on arXiv 25 Sep 2026.

Builds on

15 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates17 Jun
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 11

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  6. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  7. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  8. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  9. 2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 21
  10. 2024On the Loss of Context-awareness in General Instruction Fine-tuningcited by 10
  11. 2025Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spotscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  2. 2024Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models
  3. 2024Superficial Safety Alignment Hypothesis
  4. 2026Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation
  5. 2024Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness
  6. 2025Self-Guided Adaptive Safety Alignment: Synthesizing and Internalizing Guidelines in Reasoning Models
  7. 2024POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization
  8. 2023The Poison of Alignment