mlsanityData to 14 Oct 2026

On Evaluating the Durability of Safeguards for Open-Weight LLMs

arXiv 2412.07097 ↗First posted 10 Dec 2024cs.CR cs.AIcited by 52

Not on the map. Why?

Cited by 20 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Stakeholders -- from model developers to policymakers -- seek to minimize the dual-use risks of large language models (LLMs). An open challenge to this goal is whether technical safeguards can impede the misuse of LLMs, even when models are customizable via fine-tuning or when model weights are fully open. In response, several recent studies have proposed methods to produce durable LLM safeguards for open-weight LLMs that can withstand adversarial modifications of the model's weights via fine-tuning. This holds the promise of raising adversaries' costs even under strong threat models where adversaries can directly fine-tune model weights. However, in this paper, we urge for more careful characterization of the limits of these approaches. Through several case studies, we demonstrate that even evaluating these defenses is exceedingly difficult and can easily mislead audiences into thinking that safeguards are more durable than they really are. We draw lessons from the evaluation pitfalls that we identify and suggest future research carefully cabin claims to more constrained, well-defined, and rigorously examined threat models, which can provide more useful and candid assessments to stakeholders.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Builds on 58 of its 74 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

58 of 74 references.

References · 58

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
38 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  2. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  8. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  11. 2019Machine Unlearningcited by 1,061
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  13. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  14. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  15. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  16. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 546
  17. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  18. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  19. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  20. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  21. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  22. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  23. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  24. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  25. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 328
  26. 2023Are aligned neural networks adversarially aligned?cited by 315
  27. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  28. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  29. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  30. 2018Adversarial Risk and the Dangers of Evaluating Against Weak Attackscited by 221
  31. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  32. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  33. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  34. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 123
  35. 2024What is in Your Safe Data? Identifying Benign Data that Breaks Safetycited by 118
  36. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  37. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 80
  38. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 57

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 29; 2026: 23.

52 papers.

Sort:

By area:

  1. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  2. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  3. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  4. 2025Cyber-Zero: Training Cybersecurity Agents without Runtimecited by 22
  5. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  6. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 19
  7. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 15
  8. 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
  9. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  10. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  11. 2025Self-Destructive Language Modelcited by 10
  12. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
38 more
  1. 2025Jailbreak-Tuning: Models Efficiently Learn Jailbreak Susceptibilitycited by 9
  2. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8
  3. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 8
  4. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  5. 2025Does Machine Unlearning Truly Remove Knowledge?cited by 7
  6. 2025No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanismscited by 6
  7. 2025When Style Breaks Safety: Defending LLMs Against Superficial Style Alignmentcited by 6
  8. 2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
  9. 2025Watch your steps: Dormant Adversarial Behaviors that Activate upon LLM Finetuningcited by 5
  10. 2026Hair-Trigger Alignment: Black-Box Evaluation Cannot Guarantee Post-Update Alignmentcited by 4
  11. 2026Gauss-Newton Unlearning for the LLM Eracited by 4
  12. 2025The Backfiring Effect of Weak AI Safety Regulationcited by 3
  13. 2025Provably Safe Model Updatescited by 3
  14. 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
  15. 2025Soft Token Attacks Cannot Reliably Audit Unlearning in Large Language Modelscited by 2
  16. 2025The Model Hears You: Audio Language Model Deployments Should Consider the Principle of Least Privilegecited by 2
  17. 2025The Safety Gap Toolkit: Evaluating Hidden Dangers of Open-Source Modelscited by 2
  18. 2025HackWorld: Evaluating Computer-Use Agents on Exploiting Web Application Vulnerabilitiescited by 2
  19. 2026TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tamperingcited by 2
  20. 2026A Few Steps Further: Why Defenses Against Malicious Finetuning Erode Under Continued Trainingcited by 2
  21. 2025Training AI to be Loyalcited by 1
  22. 2025Best Practices for Biorisk Evaluations on Open-Weight Bio-Foundation Modelscited by 1
  23. 2026Steerability of Instrumental-Convergence Tendencies in LLMscited by 1
  24. 2026Security in the Fine-Tuning Lifecycle of Large Language Models: Threats, Defenses,Evaluation, and Future Directionscited by 1
  25. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attackscited by 1
  26. 2026What Does It Mean to Break a Distillation Defense?cited by 1
  27. 2026Isolation as a First-Class Principle for LLM-Agent System Safety: Concepts, Taxonomy, Challenges and Future Directionscited by 1
  28. 2026Safeguard-Conditioned Uplift: Measuring Utility-Risk Frontiers for Dual-Use Biology Assistantscited by 1
  29. 2026Distribution-Specific Curvature Control with Finite-Sample Guarantees for Open-Weight Safetycited by 1
  30. 2025BlueGlass: A Framework for Composite AI Safetycited by 0
  31. 2026A Privacy by Design Framework for Large Language Model-Based Applications for Childrencited by 0
  32. 2026Locking Pretrained Weights via Deep Low-Rank Residual Distillationcited by 0
  33. 2026Defending Against Malicious Finetuning by Scaling Train-time Adversarial Attackscited by 0
  34. 2026Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuningcited by 0
  35. 2026Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Modelscited by 0
  36. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  37. 2026Can Data Attribution Filter Out Subliminal Learning? Not Reliablycited by 0
  38. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0

50 most cited of 52 All 52 (JSON)