mlsanityData to 14 Oct 2026

Harmful Intent as a Geometrically Recoverable Feature of LLM Residual Streams

arXiv 2604.18901 ↗First posted 20 Apr 2026cs.LG cs.AI cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Aligned language models refuse harmful instructions, but the representations through which they recognise such instructions are less well characterised than the behaviours they produce. Harmful intent is linearly separable from residual-stream activations across 12 models spanning four architectural families (Qwen2.5, Qwen3.5, Llama-3.2, Gemma-3) and three alignment variants (base, instruction-tuned, abliterated), with parameter scales from 0.5B to 1.3B and a within-family scale extension to 9B on Qwen3.5. A direction fitted from 100 labelled examples per class via Soft-AUC optimisation reaches mean effective AUROC 0.982 and TPR@1%FPR 0.797, generalises to three held-out harm benchmarks and a hard-benign control, and matches its instruction-tuned counterpart within ± 0.003 AUROC in abliterated variants from which the refusal mechanism has been removed. The supervised strategies all exceed AUROC 0.96, but their TPR@1%FPR varies by more than ten times the AUROC gap; a deployed 9B safety classifier shows the same pattern at AUROC 0.94 and TPR 0.30, motivating low-FPR reporting as a default in safety-adjacent detection evaluation. Geometric measurements refine the picture. The recovered direction is concentrated within each extraction protocol but protocol-dependent across them: two pooling choices applied to the same chat-templated activations at the same residual-stream layer (max-pool over content tokens versus last-token at the post-instruction position) recover harm directions 73^circ apart, and projecting one out leaves detection under either max-pool extraction essentially intact. Probing identifies a protocol-specific direction rather than a unique computational feature.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 30 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

22 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 20

  1. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  10. 2022Toy Models of Superpositioncited by 971
  11. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  12. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
8 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  3. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  5. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 323
  6. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  7. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
  8. 2026The Geometry of Harmful Intent: Training-Free Anomaly Detection via Angular Deviation in LLM Residual Streamscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  2. 2026The Role of Fine-grained Harm Signals in LLM Safety
  3. 2026Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive Refinement
  4. 2026Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs
  5. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
  6. 2025Read the Scene, Not the Script: Outcome-Aware Safety for LLMs
  7. 2025Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs
  8. 2024No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks