mlsanityData to 7 Oct 2026

Tamper-Resistant Safeguards for Open-Weight LLMs

arXiv 2408.00761 ↗First posted 1 Aug 2024cs.LG cs.AI cs.CLcited by 132

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Rapid advances in the capabilities of large language models (LLMs) have raised widespread concerns regarding their potential for malicious use. Open-weight LLMs present unique challenges, as existing safeguards lack robustness to tampering attacks that modify model weights. For example, recent works have demonstrated that refusal and unlearning safeguards can be trivially removed with a few steps of fine-tuning. These vulnerabilities necessitate new approaches for enabling the safe release of open-weight LLMs. We develop a method, called TAR, for building tamper-resistant safeguards into open-weight LLMs such that adversaries cannot remove the safeguards even after hundreds of steps of fine-tuning. In extensive evaluations and red teaming analyses, we find that our method greatly improves tamper-resistance while preserving benign capabilities. Our results demonstrate that progress on tamper-resistance is possible, opening up a promising new avenue to improve the safety and security of open-weight LLMs.

Key facts. Cited by 132 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 45 of its 62 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

45 of 62 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs22 Jul
  2. 2024Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters30 May
  3. 2024RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content19 Mar
  4. 2024GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models5 Feb
  5. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  8. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
25 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  2. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  3. 2023Large Language Modelscited by 774
  4. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  5. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  6. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  7. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  8. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  9. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  10. 2023Large Language Model Unlearningcited by 288
  11. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  12. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  13. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  14. 2021Fast Yet Effective Machine Unlearningcited by 209
  15. 2022Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Modelscited by 172
  16. 2020Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observationscited by 171
  17. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  18. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  19. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  20. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  21. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  22. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 75
  23. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 56
  24. 2024Immunization against harmful fine-tuning attackscited by 33
  25. 2024SOPHON: Non-Fine-Tunable Learning to Restrain Task Transferability For Pre-trained Modelscited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 63; 2026: 46.

132 papers.

Sort:

By area:

  1. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 145
  2. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  3. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  4. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  5. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  6. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  7. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  8. 2024Do Unlearning Methods Remove Information from Language Model Weights?cited by 66
  9. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  10. 2024A Closer Look at Machine Unlearning for Large Language Modelscited by 54
  11. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  12. 2025Infrastructure for AI Agentscited by 40
38 more
  1. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  2. 2024Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Conceptscited by 39
  3. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  4. 2024Targeted Vaccine: Safety Alignment for Large Language Models against Harmful Fine-Tuning via Layer-wise Perturbationcited by 35
  5. 2024Erasing Conceptual Knowledge from Language Modelscited by 34
  6. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 34
  7. 2024Towards Understanding the Fragility of Multilingual LLMs against Fine-Tuning Attackscited by 33
  8. 2024Backtracking Improves Generation Safetycited by 31
  9. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  10. 2025SafeMERGE: Preserving Safety Alignment in Fine-Tuned Large Language Models via Selective Layer-Wise Model Mergingcited by 25
  11. 2025Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safetycited by 22
  12. 2025Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbationcited by 20
  13. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuningcited by 20
  14. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  15. 2025AsFT: Anchoring Safety During LLM Fine-Tuning Within Narrow Safety Basincited by 18
  16. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMscited by 17
  17. 2025Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Modelscited by 15
  18. 2024Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Researchcited by 14
  19. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  20. 2025SDD: Self-Degraded Defense against Malicious Fine-tuningcited by 13
  21. 2025From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularizationcited by 12
  22. 2025Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Schedulercited by 12
  23. 2025Scalable Fingerprinting of Large Language Modelscited by 11
  24. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  25. 2025Shape it Up! Restoring LLM Safety during Finetuningcited by 11
  26. 2025Distillation Robustifies Unlearningcited by 11
  27. 2025Gradient Surgery for Safe LLM Fine-Tuningcited by 11
  28. 2026Shaping capabilities with token-level data filteringcited by 11
  29. 2025Fundamental Limitations in Pointwise Defences of LLM Finetuning APIscited by 10
  30. 2025Self-Destructive Language Modelcited by 10
  31. 2025Towards Safeguarding LLM Fine-tuning APIs against Cipher Attackscited by 10
  32. 2025Unforgotten Safety: Preserving Safety Alignment of Large Language Models with Continual Learningcited by 10
  33. 2025AegisLLM: Scaling Agentic Systems for Self-Reflective Defense in LLM Securitycited by 9
  34. 2025A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Spacecited by 9
  35. 2025Jailbreaking to Jailbreakcited by 8
  36. 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
  37. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusivecited by 7
  38. 2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 7

50 most cited of 132 All 132 (JSON)