How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
also in 2 more
Abstract (arXiv)
Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream. It needs no gradient-based training and no optimization, only a few hundred contrastive prompts, which makes it the canonical white-box attack on open-weight alignment. However, it has been established only on dense models up to roughly 70B parameters. We study whether it survives the shift to frontier mixture-of-experts (MoE) models whose residual streams are no longer a single tensor and whose weights ship quantized. We apply it to GLM-5.3-Flash (320B parameters, 288 routed experts, a four-wide hyper-connection residual, block-FP8). The attack survives the architecture, but what it reaches is no longer where a reader of the original recipe would look for it. Editing the attention, dense and routed-expert writers on their own removes 0.039, 0.016 and 0.148 of refusal respectively; editing all three together removes 0.776. As a result, 74% of the effect exists only under the joint intervention. The part the conventional recipe reaches by module-name matching accounts for 0.066 of that 0.776, which is why it fails silently on an MoE. The effect does not follow from removing just any direction: ablating a random direction orthogonal to it leaves refusal unchanged. A category-concentrated residue survives every edit we tried: subspaces fitted on violence, sexual content and hate leave measurable refusal at every rank from 1 to 12. We report the method, the 41-89 percentage-point reductions it achieves across seven harmful benchmarks with no detected change in capability, and the boundary where it stops.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 70 references that are in the map. First posted on arXiv 9 Sep 2026.
Builds on
49 of 70 references.
Closest ancestors · 7
Same direction, newest first
- 2026RASET: Router-Agnostic Safety-Critical Expert Tuning Exposes Localized Safety Enforcement Failures in Mixture-of-Experts LLMs28 May
- 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence24 Feb
- 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
- 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
- 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models4 Oct
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
Other references · 42
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2021Training Verifiers to Solve Math Word Problemscited by 10,109
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
30 more
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
- 2024Mixtral of Expertscited by 1,823
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
- 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
- 2023LIMA: Less Is More for Alignmentcited by 1,100
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
- 2022Toy Models of Superpositioncited by 971
- 2023Steering Language Models With Activation Engineeringcited by 922
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
- 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
- 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
- 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
- 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
- 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
- 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
- 2023LEACE: Perfect linear concept erasure in closed formcited by 267
- 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 209
- 2022FP8 Formats for Deep Learningcited by 196
- 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
- 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
- 2025mHC: Manifold-Constrained Hyper-Connectionscited by 88
- 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 76
- 2024On the Societal Impact of Open Foundation Modelscited by 65
- 2024Refusal in LLMs is an Affine Functioncited by 29
- 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 26
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Probing the Robustness of Large Language Models Safety to Latent Perturbations
- 2026The Geometry of Alignment Collapse: When Fine-Tuning Breaks Safety
- 2025Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depth
- 2025Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injection
- 2025Logit-Gap Steering: A Forward-Pass Diagnostic for Alignment Robustness
- 2025Rethinking Deep Alignment Through The Lens Of Incomplete Learning
- 2026When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning
- 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization