A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models
On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks
also in 1 more
Abstract (arXiv)
Safety alignment in language models operates through two mechanistically distinct systems: refusal neurons that gate whether harmful knowledge is expressed, and concept neurons that encode the harmful knowledge itself. By targeting a single neuron in each system, we demonstrate both directions of failure -- bypassing safety on explicit harmful requests via suppression, and inducing harmful content from innocent prompts via amplification -- across seven models spanning two families and 1.7B to 70B parameters, without any training or prompt engineering. Our findings suggest that safety alignment is not robustly distributed across model weights but is mediated by individual neurons that are each causally sufficient to gate refusal behavior -- suppressing any one of the identified refusal neurons bypasses safety alignment across diverse harmful requests.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 35 references that are in the map. First posted on arXiv 8 May 2026.
Builds on
15 of 35 references.
Closest ancestors · 4
Same direction, newest first
- 2026SafeNeuron: Neuron-Level Safety Alignment for Large Language Models12 Feb
- 2025NeuroStrike: Neuron-Level Attacks on Aligned LLMs15 Sep
- 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons20 Jun
- 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
Other references · 11
- 2025Qwen3 Technical Reportcited by 8,195
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2022Toy Models of Superpositioncited by 928
- 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
- 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
- 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
- 2022What do Vision Transformers Learn? A Visual Explorationcited by 75
- 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
- 2024Jailbreak Instruction-Tuned LLMs via end-of-sentence MLP Re-weightingcited by 4
- 2025Reasoning's Razor: Reasoning Improves Accuracy but Can Hurt Recall at Critical Operating Points in Safety and Hallucination Detectioncited by 3
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026SafeEvo: Deciphering the Safety Alignment Mechanism and Evolution in Language Models
- 2026Tripwire: Triggering Aligned Refusal via Statistically Certified Safety Neurons
- 2026NeuronGuard: Robust LLM Safety Alignment via Ablation-Aware Safety Signal Redistribution
- 2025NeuronTune: Fine-Grained Neuron Modulation for Balanced Safety-Utility Alignment in LLMs
- 2026Who Transfers Safety? Identifying and Targeting Cross-Lingual Shared Safety Neurons
- 2025NeuRel-Attack: Neuron Relearning for Safety Disalignment in Large Language Models
- 2026When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning
- 2026No Single Neuron of Failure: Distributed Safety Alignment Against White-Box Attacks