mlsanityData to 14 Oct 2026

Refusal in Language Models Is Mediated by a Single Direction

arXiv 2406.11717 ↗First posted 17 Jun 2024cs.LG cs.AI cs.CLcited by 1,032

On the map: LLM safety and security → Safety alignment, refusal and guardrails

Abstract (arXiv)

Conversational large language models are fine-tuned for both instruction-following and safety, resulting in models that obey benign requests but refuse harmful ones. While this refusal behavior is widespread across chat models, its underlying mechanisms remain poorly understood. In this work, we show that refusal is mediated by a one-dimensional subspace, across 13 popular open-source chat models up to 72B parameters in size. Specifically, for each model, we find a single direction such that erasing this direction from the model's residual stream activations prevents it from refusing harmful instructions, while adding this direction elicits refusal on even harmless instructions. Leveraging this insight, we propose a novel white-box jailbreak method that surgically disables refusal with minimal effect on other capabilities. Finally, we mechanistically analyze how adversarial suffixes suppress propagation of the refusal-mediating direction. Our findings underscore the brittleness of current safety fine-tuning methods. More broadly, our work showcases how an understanding of model internals can be leveraged to develop practical methods for controlling model behavior.

Key facts. Cited by 1,032 papers in the map (data through 14 Oct 2026). Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security. Builds on 72 of its 102 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

72 of 102 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  2. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70B31 Oct
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct

Other references · 69

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  11. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  12. 2023Qwen Technical Reportcited by 3,344
38 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  9. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  11. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  12. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  13. 2022Red Teaming Language Models with Language Modelscited by 1,026
  14. 2022Toy Models of Superpositioncited by 971
  15. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  16. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  17. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  18. 2024Yi: Open Foundation Models by 01.AIcited by 719
  19. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  20. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  21. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  22. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  23. 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 469
  24. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  25. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  26. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  27. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  28. 2024A StrongREJECT for Empty Jailbreakscited by 385
  29. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  30. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  31. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  32. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 297
  33. 2023Role-Play with Large Language Modelscited by 294
  34. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  35. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 214
  36. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  37. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  38. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 59; 2025: 315; 2026: 658.

1,032 papers.

Sort:

By area:

  1. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  2. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  3. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  4. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yetcited by 151
  5. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 123
  6. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 122
  7. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 118
  8. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 115
  9. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 115
  10. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 95
  11. 2025Model Organisms for Emergent Misalignmentcited by 91
  12. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 88
38 more
  1. 2025Persona Features Control Emergent Misalignmentcited by 86
  2. 2024On the Role of Attention Heads in Large Language Model Safetycited by 72
  3. 2025Understanding Reasoning in Thinking Language Models via Steering Vectorscited by 72
  4. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 71
  5. 2025Convergent Linear Representations of Emergent Misalignmentcited by 71
  6. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 64
  7. 2024Benchmarking Distributional Alignment of Large Language Modelscited by 61
  8. 2024BEEAR: Embedding-based Adversarial Removal of Safety Backdoors in Instruction-tuned Language Modelscited by 58
  9. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 58
  10. 2024What Makes and Breaks Safety Fine-tuning? A Mechanistic Studycited by 55
  11. 2025JBShield: Defending Large Language Models from Jailbreak Attacks through Activated Concept Analysis and Manipulationcited by 54
  12. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 53
  13. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 53
  14. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 52
  15. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 51
  16. 2024ICLR: In-Context Learning of Representationscited by 50
  17. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 49
  18. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 49
  19. 2024Surgical, Cheap, and Flexible: Mitigating False Refusal in Language Models via Single Vector Ablationcited by 48
  20. 2024Persistent Pre-Training Poisoning of LLMscited by 48
  21. 2024Steering Away from Harm: An Adaptive Approach to Defending Vision Language Model Against Jailbreakscited by 48
  22. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  23. 2025Understanding (Un)Reliability of Steering Vectors in Language Modelscited by 47
  24. 2024Steering Without Side Effects: Improving Post-Deployment Control of Language Modelscited by 46
  25. 2024Language Model Alignment in Multilingual Trolley Problemscited by 46
  26. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 46
  27. 2025LLM Unlearning via Neural Activation Redirectioncited by 41
  28. 2025The Hidden Dimensions of LLM Alignment: A Multi-Dimensional Analysis of Orthogonal Safety Directionscited by 41
  29. 2025Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activationscited by 41
  30. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  31. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 40
  32. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 39
  33. 2025AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defendercited by 38
  34. 2025Refusal Direction is Universal Across Safety-Aligned Languagescited by 38
  35. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 37
  36. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 35
  37. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  38. 2025Probing and Steering Evaluation Awareness of Language Modelscited by 35

50 most cited of 1,032 All 1,032 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Optimizing Against Safety Representations: Activation-Guided Adversarial Suffixes and the Geometry of Refusal
  2. 2026Do Language Models Know When They'll Refuse? Probing Introspective Awareness of Safety Boundaries
  3. 2023I'm Afraid I Can't Do That: Predicting Prompt Refusal in Black-Box Generative Language Models
  4. 2025Where Do Reasoning Models Refuse?
  5. 2026Evaluating Language Model Safety Across Long Adversarial Conversations
  6. 2025Beyond Over-Refusal: Scenario-Based Diagnostics and Post-Hoc Mitigation for Exaggerated Refusals in LLMs
  7. 2026How User-AI Mistreatment Occurs and Matters in Conversational Systems?
  8. 2024The Art of Saying No: Contextual Noncompliance in Language Models