mlsanityData to 14 Oct 2026

How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE

arXiv 2609.09793 ↗First posted 9 Sep 2026cs.CR cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 2 more
Abstract (arXiv)

Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream. It needs no gradient-based training and no optimization, only a few hundred contrastive prompts, which makes it the canonical white-box attack on open-weight alignment. However, it has been established only on dense models up to roughly 70B parameters. We study whether it survives the shift to frontier mixture-of-experts (MoE) models whose residual streams are no longer a single tensor and whose weights ship quantized. We apply it to GLM-5.3-Flash (320B parameters, 288 routed experts, a four-wide hyper-connection residual, block-FP8). The attack survives the architecture, but what it reaches is no longer where a reader of the original recipe would look for it. Editing the attention, dense and routed-expert writers on their own removes 0.039, 0.016 and 0.148 of refusal respectively; editing all three together removes 0.776. As a result, 74% of the effect exists only under the joint intervention. The part the conventional recipe reaches by module-name matching accounts for 0.066 of that 0.776, which is why it fails silently on an MoE. The effect does not follow from removing just any direction: ablating a random direction orthogonal to it leaves refusal unchanged. A category-concentrated residue survives every edit we tried: subspaces fitted on violence, sexual content and hate leave measurable refusal at every rank from 1 to 12. We report the method, the 41-89 percentage-point reductions it achieves across seven harmful benchmarks with no detected change in capability, and the boundary where it stops.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 70 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

49 of 70 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026RASET: Router-Agnostic Safety-Critical Expert Tuning Exposes Localized Safety Enforcement Failures in Mixture-of-Experts LLMs28 May
  2. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence24 Feb
  3. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  4. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  6. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
30 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  3. 2024Mixtral of Expertscited by 1,823
  4. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  5. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  8. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  9. 2023LIMA: Less Is More for Alignmentcited by 1,100
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  11. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  12. 2022Toy Models of Superpositioncited by 971
  13. 2023Steering Language Models With Activation Engineeringcited by 922
  14. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  15. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  16. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  17. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  18. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  19. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  20. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  21. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  22. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
  23. 2022FP8 Formats for Deep Learningcited by 196
  24. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  25. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  26. 2025mHC: Manifold-Constrained Hyper-Connectionscited by 88
  27. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 76
  28. 2024On the Societal Impact of Open Foundation Modelscited by 65
  29. 2024Refusal in LLMs is an Affine Functioncited by 29
  30. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 26

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Probing the Robustness of Large Language Models Safety to Latent Perturbations
  2. 2026The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety
  3. 2025Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth
  4. 2025Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection
  5. 2025Logit-Gap Steering: A Forward-Pass Diagnostic for Alignment Robustness
  6. 2025Rethinking Deep Alignment Through The Lens Of Incomplete Learning
  7. 2026When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning
  8. 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization