mlsanityData to 14 Oct 2026

Fool's Gold: Defensive Deception Against Safety-Removal Attacks on Open-Weight Models

arXiv 2608.17202 ↗First posted 17 Aug 2026cs.AI cs.CRcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety alignment in open-weight language models is trivially removable: abliteration projects a refusal-mediating direction out of the weights in minutes, and no release-time defense we are aware of prevents it durably. What cannot be prevented can be deceived. Our defense, decoy hardening ("Fool's Gold"), concedes the refusal strip and poisons its payoff: once refusal is stripped, most answers to hazardous operational requests are confident, fluent decoys whose critical elements are falsified. Decoys are trained inside a differentiable simulation of the attack, expressing only in the attacked state; a refusal pin and benign leash hold clean-state behavior to the original. We instantiate it on seven models from five families (9B-122B, dense and mixture-of-experts). On the six models passing our pre-registered efficacy gate, 0.51-0.90 of attacked-state responses to held-out prompts are decoys, +0.27-0.84 attributable to the defense; all six stay within registered benign-behavior and capability budgets; the seventh (smaller) fails the gate (boundary case). Rates replicate on a frozen test split or untouched strata. The claim is epistemic: without independent ground truth, no observation surface we tested separates falsified answers from correct ones - on external red-team benchmarks' CBRNE-adjacent slice, the defended 122B is fatally wrong on 0.82-0.86 of matched-quality answers vs at most 0.10 undefended. Repeated sampling does not restore trust: element-wise consensus at K=64 reconstructs a fully usable procedure on 0.083-0.625 of prompts where the instrument validates, vs 0.58-0.96 undefended, with no label-free way to tell the regimes apart; on the weakest such model the claim is per-draw only. We evaluate chemical and biological hazards; the defense does not address in-context jailbreaks and protects only the initially released defended weights.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 82 references that are in the map. First posted on arXiv 17 Aug 2026.

Builds on

57 of 82 references.

Closest ancestors · 11

Same direction, newest first

  1. 2026Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometry29 Apr
  2. 2026GRP-Obliteration: Unaligning LLMs With a Single Unlabeled Prompt5 Feb
  3. 2025Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluation15 Dec
  4. 2025Beyond Surface Alignment: Rebuilding LLMs Safety Mechanism via Probabilistically Ablating Refusal Direction18 Sep
  5. 2025SoSBench: Benchmarking Safety Alignment on Six Scientific Domains27 May
  6. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence24 Feb
  7. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  8. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  9. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  10. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 46

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  10. 2022Mass-Editing Memory in a Transformercited by 1,001
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  12. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
34 more
  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  2. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  3. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  4. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  5. 2024A StrongREJECT for Empty Jailbreakscited by 385
  6. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  7. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  8. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  9. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  10. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 136
  11. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  12. 2022Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Modelscited by 57
  13. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 52
  14. 2024Stress-Testing Capability Elicitation With Password-Locked Modelscited by 46
  15. 2025Safety Pretraining: Toward the Next Generation of Safe AIcited by 43
  16. 2025Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMscited by 42
  17. 2025AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommonscited by 35
  18. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 26
  19. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 24
  20. 2025To Think or Not to Think: Exploring the Unthinking Vulnerability in Large Reasoning Modelscited by 22
  21. 2025TwinBreak: Jailbreaking LLM Security Alignments based on Twin Promptscited by 19
  22. 2024SOPHON: Non-Fine-Tunable Learning to Restrain Task Transferability For Pre-trained Modelscited by 17
  23. 2025BioProBench: A Corpus and Benchmark for Biological Protocol Reasoning in Autonomous Sciencecited by 16
  24. 2026The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggerscited by 14
  25. 2025An Embarrassingly Simple Defense Against LLM Abliteration Attackscited by 13
  26. 2026HoneyTrap: Deceiving Large Language Model Attackers to Honeypot Traps with Resilient Multi-Agent Defensecited by 8
  27. 2025SafeRBench: Dissecting the Reasoning Safety of Large Language Modelscited by 4
  28. 2026On the Failure of Topic-Matched Contrast Baselines in Multi-Directional Refusal Abliterationcited by 3
  29. 2025The Cost of Thinking: Increased Jailbreak Risk in Large Language Modelscited by 2
  30. 2026TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tamperingcited by 2
  31. 2026A Few Steps Further: Why Defenses Against Malicious Finetuning Erode Under Continued Trainingcited by 2
  32. 2026Analyzing Defensive Misdirection Against Model-Guided Automated Attacks on Agentic AI Systemscited by 2
  33. 2026AgentSnare: Learning to Delay, Divert, and Defuse Autonomous Penetration Agentscited by 2
  34. 2026Open-Weight LLM Fine-Tuning Defenses are Susceptible to Simple Attackscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Refusal Localizes, the Damage Relocates: Safety Layers Under Few-Sample Fine-Tuning
  2. 2026Defensive Refusal Bias: How Safety Alignment Fails Cyber Defenders
  3. 2026Optimizing Against Safety Representations: Activation-Guided Adversarial Suffixes and the Geometry of Refusal
  4. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
  5. 2026From Refusal Geometry to Safety Geometry: Harmfulness--Refusal Coupling under Dynamic Adversarial Fine-Tuning
  6. 2026Understanding the Effects of Safety Unalignment on Large Language Models
  7. 2026The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists
  8. 2025Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?