mlsanityData to 7 Oct 2026

Understanding Refusal in Language Models with Sparse Autoencoders

arXiv 2505.23556 ↗First posted 29 May 2025cs.CLcited by 26

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

Refusal is a key safety behavior in aligned language models, yet the internal mechanisms driving refusals remain opaque. In this work, we conduct a mechanistic study of refusal in instruction-tuned LLMs using sparse autoencoders to identify latent features that causally mediate refusal behaviors. We apply our method to two open-source chat models and intervene on refusal-related features to assess their influence on generation, validating their behavioral impact across multiple harmful datasets. This enables a fine-grained inspection of how refusal manifests at the activation level and addresses key research questions such as investigating upstream-downstream latent relationship and understanding the mechanisms of adversarial jailbreaking techniques. We also establish the usefulness of refusal features in enhancing generalization for linear probes to out-of-distribution adversarial samples in classification tasks. We open source our code in https://github.com/wj210/refusal_sae.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 16 of its 37 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

16 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  8. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  9. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  10. 2023Steering Language Models With Activation Engineeringcited by 824
  11. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  12. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
2 more
  1. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  2. 2024GASP: Efficient Black-Box Generation of Adversarial Suffixes for Jailbreaking LLMscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 20.

26 papers.

Sort:

By area:

  1. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  2. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 22
  3. 2026There Is More to Refusal in Large Language Models than a Single Directioncited by 22
  4. 2026Do Sparse Autoencoders Identify Reasoning Features in Language Models?cited by 11
  5. 2025SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Modelscited by 10
  6. 2026Steered LLM Activations are Non-Surjectivecited by 7
  7. 2025Mitigating Jailbreaks with Intent-Aware LLMscited by 4
  8. 2026Large Language Models Generate Harmful Responses Using a Distinct Mechanism, Shared Across Harm Typescited by 3
  9. 2026Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Stalenesscited by 3
  10. 2025Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptationcited by 2
  11. 2026OpenSafeIntent: Evaluating Intent-Calibrated Safe Completion Across Dual-Use Prompt Setscited by 2
  12. 2025Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMscited by 1
14 more
  1. 2025Graph-Regularized Sparse Autoencoders for LLM Safety Steeringcited by 1
  2. 2026Over-Refusal and Representation Subspaces: A Mechanistic Analysis of Task-Conditioned Refusal in Aligned LLMscited by 1
  3. 2026CRaFT: Circuit-Guided Refusal Feature Selection via Cross-Layer Transcoderscited by 1
  4. 2026Robust Harmful Features Under Jailbreak Attacks: Mechanistic Evidence from Attention Head Specialization in Large Language Modelscited by 1
  5. 2026Minimal, Local, Causal Explanations for Jailbreak Success in Large Language Modelscited by 0
  6. 2026Beyond "I cannot fulfill this request": Alleviating Rigid Rejection in LLMs via Label Enhancementcited by 0
  7. 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-offcited by 0
  8. 2026SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behaviorcited by 0
  9. 2026ReCal3R: Reliability-Calibrated Learning Rates for Streaming 3D Reconstructioncited by 0
  10. 2026Do LLMs Know Their Vulnerable Scenarios?cited by 0
  11. 2026Broken Symmetry in LLM Refusal: Answer Release Is More Local Than Refusal Restorationcited by 0
  12. 2026The Safety Relay in Roleplay Jailbreaks: A Component-Resolved Causal Analysis of Harm Recognition and Refusalcited by 0
  13. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustnesscited by 0
  14. 2026Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliterationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Refusal geometry reflects refusal training: diverse refusal prefixes can raise stable rank and weaken refusal vector ablation attacks
  2. 2026Latent-space Attacks for Refusal Evasion in Language Models
  3. 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence
  4. 2026EmoRSS: Mitigating Emotion-Induced Over-Refusal in Large Language Models
  5. 2024Cannot or Should Not? Automatic Analysis of Refusal Composition in IFT/RLHF Datasets and Refusal Behavior of Black-Box LLMs
  6. 2026Tool Mediation Alters Refusal Mechanisms in Large Language Models
  7. 2025SOM Directions are Better than One: Multi-Directional Refusal Suppression in Language Models
  8. 2025RefusalBench: Generative Evaluation of Selective Refusal in Grounded Language Models