mlsanityData to 7 Oct 2026

Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation

arXiv 2406.20053 ↗First posted 28 Jun 2024cs.CR cs.AI cs.CLcited by 74

Not on the map. Why?

Cited by 39 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Black-box finetuning is an emerging interface for adapting state-of-the-art language models to user needs. However, such access may also let malicious actors undermine model safety. To demonstrate the challenge of defending finetuning interfaces, we introduce covert malicious finetuning, a method to compromise model safety via finetuning while evading detection. Our method constructs a malicious dataset where every individual datapoint appears innocuous, but finetuning on the dataset teaches the model to respond to encoded harmful requests with encoded harmful responses. Applied to GPT-4, our method produces a finetuned model that acts on harmful instructions 99% of the time and avoids detection by defense mechanisms such as dataset inspection, safety evaluations, and input/output classifiers. Our findings question whether black-box finetuning access can be secured against sophisticated adversaries.

Key facts. Cited by 74 papers in the map (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

26 of 36 references.

References · 26

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2023Let's Verify Step by Stepcited by 3,598
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  12. 2022Red Teaming Language Models with Language Modelscited by 983
14 more
  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  2. 2023Instruction Tuning with GPT-4cited by 676
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2020Backdoor Learning: A Surveycited by 412
  5. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  6. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  7. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  8. 2023Poisoning Language Models During Instruction Tuningcited by 254
  9. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  10. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 150
  11. 2023On the Exploitability of Instruction Tuningcited by 110
  12. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  13. 2023Exploiting Novel GPT-4 APIscited by 27
  14. 2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 32; 2026: 27.

74 papers.

Sort:

By area:

  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  2. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  3. 2025Agentic Misalignment: How LLMs Could Be Insider Threatscited by 136
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  5. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  7. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  8. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  9. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 67
  10. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 66
  11. 2024Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMscited by 52
  12. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
38 more
  1. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  2. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  3. 2024Scaling Trends for Data Poisoning in LLMscited by 26
  4. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  5. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  6. 2025How to evaluate control measures for LLM agents? A trajectory from today to superintelligencecited by 20
  7. 2024Endless Jailbreaks with Bijection Learningcited by 18
  8. 2025Why LLM Safety Guardrails Collapse After Fine-tuning: A Similarity Analysis Between Alignment and Fine-tuning Datasetscited by 18
  9. 2025A Survey on Model Extraction Attacks and Defenses for Large Language Modelscited by 18
  10. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17
  11. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17
  12. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 15
  13. 2026Subliminal Effects in Your Data: A General Mechanism via Log-Linearitycited by 15
  14. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  15. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  16. 2025Self-Destructive Language Modelcited by 10
  17. 2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 10
  18. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
  19. 2026Your LLM Agent Can Leak Your Data: Data Exfiltration via Backdoored Tool Usecited by 9
  20. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  21. 2026Introspection Adapters: Training LLMs to Report Their Learned Behaviorscited by 8
  22. 2025LookAhead Tuning: Safer Language Models via Partial Answer Previewscited by 7
  23. 2025All Code, No Thought: Current Language Models Struggle to Reason in Ciphered Languagecited by 7
  24. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 6
  25. 2024Recent Advances in Attack and Defense Approaches of Large Language Modelscited by 5
  26. 2024The Tug of War Within: Mitigating the Fairness-Privacy Conflicts in Large Language Modelscited by 5
  27. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 5
  28. 2025SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requestscited by 5
  29. 2025Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuningcited by 4
  30. 2024Towards Data Governance of Frontier AI Modelscited by 3
  31. 2025AI Companies Should Report Pre- and Post-Mitigation Safety Evaluationscited by 3
  32. 2025JULI: Jailbreak Large Language Models by Self-Introspectioncited by 3
  33. 2025OneShield -- the Next Generation of LLM Guardrailscited by 3
  34. 2025Detecting Adversarial Fine-tuning with Auditing Agentscited by 3
  35. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  36. 2026Segment-Level Coherence for Robust Harmful Intent Probing in LLMscited by 3
  37. 2026AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMscited by 2
  38. 2026Invisible Safety Threat: Malicious Finetuning for LLM via Steganographycited by 2

50 most cited of 74 All 74 (JSON)