mlsanityData to 14 Oct 2026

MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generation

arXiv 2510.07835 ↗First posted 9 Oct 2025cs.LG cs.AI cs.CLcited by 8

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

This paper introduces MetaDefense, a novel framework for defending against finetuning-based jailbreak attacks in large language models (LLMs). We observe that existing defense mechanisms fail to generalize to harmful queries disguised by unseen attack templates, despite LLMs being capable of distinguishing disguised harmful queries in the embedding space. Based on these insights, we propose a two-stage defense approach: (i) pre-generation defense that detects harmful queries before response generation begins, and (ii) mid-generation defense that monitors partial responses during generation to prevent outputting more harmful content. Our MetaDefense trains the LLM to predict the harmfulness of both queries and partial responses using specialized prompts, enabling early termination of potentially harmful interactions. Extensive experiments across multiple LLM architectures (LLaMA-2-7B, Qwen-2.5-3B-Instruct, and LLaMA-3.2-3B-Instruct) demonstrate that MetaDefense significantly outperforms existing defense mechanisms, achieving robust defense against harmful queries with seen and unseen attack templates while maintaining competitive performance on benign tasks. Code is available at https://github.com/ws-jiang/MetaDefense.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 45 of its 57 references that are in the map. First posted on arXiv 9 Oct 2025.

Builds on

45 of 57 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreaks13 Dec
  2. 2024RobustKV: Defending Large Language Models against Jailbreak Attacks via KV Eviction25 Oct
  3. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusal25 Apr
  4. 2023Exploiting Novel GPT-4 APIs21 Dec
  5. 2023Jailbreaking GPT-4V via Self-Adversarial Attacks with System Prompts15 Nov
  6. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  7. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov

Other references · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  10. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  12. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
26 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  4. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  5. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  6. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  7. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  8. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 125
  9. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 115
  10. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 96
  11. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 96
  12. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  13. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
  14. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Modelscited by 59
  15. 2024A Theoretical Understanding of Self-Correction through In-context Alignmentcited by 44
  16. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  17. 2025SaLoRA: Safety-Alignment Preserved Low-Rank Adaptationcited by 42
  18. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 40
  19. 2024Immunization against harmful fine-tuning attackscited by 35
  20. 2024GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoningcited by 33
  21. 2024Backtracking Improves Generation Safetycited by 32
  22. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 28
  23. 2024MTMamba: Enhancing Multi-Task Dense Scene Understanding by Mamba-Based Decoderscited by 22
  24. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 16
  25. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14
  26. 2024MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoderscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025SoK: Systematizing LLM Prompt Security: Taxonomies, Datasets, and Unified Evaluation of Attacks and Defensescited by 5
  2. 2025Unified Defense for Large Language Models against Jailbreak and Fine-Tuning Attacks in Educationcited by 5
  3. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
  4. 2026THRD: A Training-Free Multi-Turn Defense Framework for Jailbreak Attacks on Large Language Modelscited by 2
  5. 2026DataShield: Safety-degrading Data Filtering for LLM Benign Instruction Fine-Tuningcited by 1
  6. 2026MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unificationcited by 0
  7. 2026SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selectioncited by 0
  8. 2026Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Retrieval-Augmented Defense: Adaptive and Controllable Jailbreak Prevention for Large Language Models
  2. 2025Scalable Defense against In-the-wild Jailbreaking Attacks with Safety Context Retrieval
  3. 2026CASCADE Against Jailbreaks: Combination Across Stages with Controlled Attack-Defense Evaluation
  4. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  5. 2025Defending Large Language Models Against Jailbreak Exploits with Responsible AI Considerations
  6. 2024Prefix Guidance: A Steering Wheel for Large Language Models to Defend Against Jailbreak Attacks
  7. 2025Fine-Tuning Jailbreaks under Highly Constrained Black-Box Settings: A Three-Pronged Approach
  8. 2024Leveraging the Context through Multi-Round Interactions for Jailbreaking Attacks