Data to 14 Oct 2026

Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Survey

arXiv 2409.18169 ↗First posted 26 Sep 2024cs.CR cs.AI cs.LGcited by 90

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more

LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Recent research demonstrates that the nascent fine-tuning-as-a-service business model exposes serious safety concerns: fine-tuning with a few harmful data uploaded from the users can compromise the safety alignment of the model. The attack, known as harmful fine-tuning attack, has generated broad research interests in both academia and industry. In this paper, we first systematically formulate the threat model and basic assumptions of harmful fine-tuning. Then, we provide a comprehensive review of harmful fine-tuning from three fundamental perspectives: attack setting, defense design, and evaluation methodology. First, we present the threat model of the problem and introduce the harmful fine-tuning attack and its variants. Next, we systematically survey representative attacks, defense methods, and mechanical analysis of adverse effects in the existing literature. Finally, we introduce the evaluation methodology and outline future research directions, which can serve as guidelines and crucial perspectives for the future development of the subject. We also maintain a curated list of relevant papers, which are made accessible at https://github.com/git-disl/awesome_LLM-harmful-fine-tuning-papers

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 137 of its 285 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

137 of 285 references.

Closest ancestors · 21

Same direction, newest first

9 more

Other references · 116

2022Training language models to follow instructions with human feedbackcited by 15,971
2017Proximal Policy Optimization Algorithmscited by 15,215
2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
2020Measuring Massive Multitask Language Understandingcited by 7,963
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720

38 more

2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
2015Character-level Convolutional Networks for Text Classificationcited by 2,330
2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
2020Extracting Training Data from Large Language Modelscited by 2,189
2024Qwen2 Technical Reportcited by 2,120
2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
2017BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chaincited by 1,106
2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
2018How To Backdoor Federated Learningcited by 765
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 535
2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
2018Spectral Signatures in Backdoor Attackscited by 484
2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 379
2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
2024Improving Alignment and Robustness with Circuit Breakerscited by 322
2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
2020Attack of the Tails: Yes, You Really Can Backdoor Federated Learningcited by 309
2017Certified Defenses for Data Poisoning Attackscited by 303
2023Preference Ranking Optimization for Human Alignmentcited by 285
2020Data Poisoning Attacks Against Federated Learning Systemscited by 275

50 most cited of 116 All 116 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 60; 2026: 27.

90 papers.

Sort:

By area:

2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 122
2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 62
2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuningcited by 37
2025Backdoor Cleaning without External Guidance in MLLM Fine-tuningcited by 35
2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 20
2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 19
2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 19

38 more

2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 19
2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 18
2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
2025Fundamental Safety-Capability Trade-offs in Fine-tuning Large Language Modelscited by 17
2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 15
2024Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directionscited by 14
2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 13
2025SDD: Self-Degraded Defense against Malicious Fine-tuningcited by 13
2025LoX: Low-Rank Extrapolation Robustifies LLM Safety Against Fine-tuningcited by 12
2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 12
2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 11
2025Think Twice, Generate Once: Safeguarding by Progressive Self-Reflectioncited by 11
2025Single-pass Detection of Jailbreaking Input in Large Language Modelscited by 10
2025Self-Destructive Language Modelcited by 10
2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 9
2025Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuningcited by 9
2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
2025Model Immunization from a Condition Number Perspectivecited by 8
2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 8
2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defensecited by 7
2025Toward Robust Non-Transferable Learning: A Survey and Benchmarkcited by 7
2025Anchoring Refusal Direction: Mitigating Safety Risks in Tuning via Projection Constraintcited by 7
2026GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Promptcited by 7
2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 6
2025Safety Subspaces are Not Linearly Distinct: A Fine-Tuning Case Studycited by 6
2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
2025Pharmacist: Safety Alignment Data Curation for Large Language Models against Harmful Fine-tuningcited by 6
2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
2025LLM Misalignment via Adversarial RLHF Platformscited by 5
2025Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuningcited by 5
2025Accidental Vulnerability: Factors in Fine-Tuning that Shift Model Safeguardscited by 5

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way