mlsanityData to 7 Oct 2026

The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence

arXiv 2502.17420 ↗First posted 24 Feb 2025cs.LG cs.AI cs.CLcited by 104

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

The safety alignment of large language models (LLMs) can be circumvented through adversarially crafted inputs, yet the mechanisms by which these attacks bypass safety barriers remain poorly understood. Prior work suggests that a single refusal direction in the model's activation space determines whether an LLM refuses a request. In this study, we propose a novel gradient-based approach to representation engineering and use it to identify refusal directions. Contrary to prior work, we uncover multiple independent directions and even multi-dimensional concept cones that mediate refusal. Moreover, we show that orthogonality alone does not imply independence under intervention, motivating the notion of representational independence that accounts for both linear and non-linear effects. Using this framework, we identify mechanistically independent refusal directions. We show that refusal mechanisms in LLMs are governed by complex spatial structures and identify functionally independent directions, confirming that multiple distinct mechanisms drive refusal behavior. Our gradient-based approach uncovers these mechanisms and can further serve as a foundation for future work on understanding LLMs.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 58 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

46 of 58 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directions13 Feb
  2. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Models1 Sep
  3. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Models31 May
  4. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  5. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space14 Feb
  6. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 39

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2013Intriguing properties of neural networkscited by 5,114
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  11. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
27 more
  1. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  4. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  7. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  8. 2024A StrongREJECT for Empty Jailbreakscited by 352
  9. 2023Language Models Represent Space and Timecited by 306
  10. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  11. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  12. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  13. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 224
  14. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  15. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  16. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  17. 2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
  18. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  19. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  20. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 51
  21. 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreakscited by 45
  22. 2024Revisiting Jailbreaking for Large Language Models: A Representation Engineering Perspectivecited by 44
  23. 2024Navigating the OverKill in Large Language Modelscited by 40
  24. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  25. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  26. 2024Monotonic Representation of Numeric Properties in Language Modelscited by 15
  27. 2025Adversarial Alignment for LLMs Requires Simpler, Reproducible, and More Measurable Objectivescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 85.

104 papers.

Sort:

By area:

  1. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  2. 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
  3. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  4. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  5. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
  6. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
  7. 2026Sycophancy Hides Linearly in the Attention Headscited by 21
  8. 2025Beyond I'm Sorry, I Can't: Dissecting Large Language Model Refusalcited by 17
  9. 2025On Reasoning Strength Planning in Large Reasoning Modelscited by 13
  10. 2025RepIt: Steering Language Models with Concept-Specific Refusal Vectorscited by 10
  11. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 9
  12. 2025SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Modelscited by 9
38 more
  1. 2026The Information Geometry of Softmax: Probing and Steeringcited by 9
  2. 2026Analysing the Safety Pitfalls of Steering Vectorscited by 9
  3. 2025Comparative Analysis of LLM Abliteration Methods: A Cross-Architecture Evaluationcited by 8
  4. 2026Why Steering Works: Toward a Unified View of Language Model Parameter Dynamicscited by 8
  5. 2025The Geometry of Harmfulness in LLMs through Subconcept Probingcited by 7
  6. 2025Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learningcited by 7
  7. 2026What Drives Representation Steering? A Mechanistic Case Study on Steering Refusalcited by 7
  8. 2025Sysformer: Safeguarding Frozen Large Language Models with Adaptive System Promptscited by 5
  9. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  10. 2026Curveball Steering: The Right Direction To Steer Isn't Always Linearcited by 5
  11. 2025From Directions to Cones: Exploring Multidimensional Representations of Propositional Facts in LLMscited by 3
  12. 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
  13. 2025Where Do Reasoning Models Refuse?cited by 3
  14. 2026Conditioned Activation Transport for T2I Safety Steeringcited by 3
  15. 2026On the Failure of Topic-Matched Contrast Baselines in Multi-Directional Refusal Abliterationcited by 3
  16. 2026How Alignment Routes: Localizing, Scaling, and Controlling Policy Circuits in Language Modelscited by 3
  17. 2026Before the Last Token: Diagnosing Final-Token Safety Probe Failurescited by 3
  18. 2025Any-Depth Alignment: Unlocking Innate Safety Alignment of LLMs to Any-Depthcited by 2
  19. 2026Fail-Closed Alignment for Large Language Modelscited by 2
  20. 2026SafeDream: Safety World Model for Proactive Early Jailbreak Detectioncited by 2
  21. 2026Decodable but Not Corrected by Fixed Residual-Stream Linear Steering: Evidence from Medical LLM Failure Regimescited by 2
  22. 2026Low-Resource Safety Failures Are Action Failures, Not Representation Failurescited by 2
  23. 2026Controlling Tool Use with Heading-Specific Activation Steeringcited by 2
  24. 2026The Entanglement Wall: Activation-Space Probes as Risk Detectors, Not Context Adjudicatorscited by 2
  25. 2025Graph-Regularized Sparse Autoencoders for LLM Safety Steeringcited by 1
  26. 2026Steerability of Instrumental-Convergence Tendencies in LLMscited by 1
  27. 2026Step-Wise Refusal Dynamics in Autoregressive and Diffusion Language Modelscited by 1
  28. 2026Discovering Universal Activation Directions for PII Leakage in Language Modelscited by 1
  29. 2026Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMscited by 1
  30. 2026Persona Non Grata: Single-Method Safety Evaluation Is Incomplete for Persona-Imbued LLMscited by 1
  31. 2026Dynamic Adversarial Fine-Tuning Reorganizes Refusal Geometrycited by 1
  32. 2026RefusalGuard: Geometry-Preserving Fine-Tuning for Safety in LLMscited by 1
  33. 2026Latent-space Attacks for Refusal Evasion in Language Modelscited by 1
  34. 2026Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Modelscited by 1
  35. 2026Not All Refusals Are Equal: How Safety Alignment Fails Cybersecurity at Scalecited by 1
  36. 2026Capsule Lens: Locating and Tracking Concept Geometry in Model Representationscited by 1
  37. 2025MEUV: Achieving Fine-Grained Capability Activation in Large Language Models via Mutually Exclusive Unlock Vectorscited by 0
  38. 2026Attributing and Exploiting Safety Vectors through Global Optimization in Large Language Modelscited by 0

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Geometry of Refusal: Linear Instability in Safety-Aligned LLMs
  2. 2025Refusal Direction is Universal Across Safety-Aligned Languages
  3. 2025Understanding Refusal in Language Models with Sparse Autoencoders
  4. 2026Efficient Refusal Ablation in LLM through Optimal Transport
  5. 2026The Geometry of Harmfulness in Multi-Turn Attacks
  6. 2026From Refusal Geometry to Safety Geometry: Harmfulness--Refusal Coupling under Dynamic Adversarial Fine-Tuning
  7. 2026Refusal Localizes, the Damage Relocates: Safety Layers Under Few-Sample Fine-Tuning
  8. 2026On the Steering Dimensionality of Refusal in Language Models