mlsanityData to 7 Oct 2026

SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steering

arXiv 2408.11491 ↗First posted 21 Aug 2024cs.AIcited by 45

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Safety alignment is indispensable for Large Language Models (LLMs) to defend threats from malicious instructions. However, recent researches reveal safety-aligned LLMs prone to reject benign queries due to the exaggerated safety issue, limiting their helpfulness. In this paper, we propose a Safety-Conscious Activation Steering (SCANS) method to mitigate the exaggerated safety concerns in aligned LLMs. First, SCANS extracts the refusal steering vectors within the activation space and utilizes vocabulary projection to anchor some specific safety-critical layers which influence model refusal behavior. Second, by tracking the hidden state transition, SCANS identifies the steering direction and steers the model behavior accordingly, achieving a balance between exaggerated safety and adequate safety. Experiments show that SCANS achieves new state-of-the-art performance on XSTest and OKTest benchmarks, without impairing their defense capability against harmful queries and maintaining almost unchanged model capability.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 33 references that are in the map. First posted on arXiv 21 Aug 2024.

Builds on

22 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 21

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  10. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  11. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  12. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
9 more
  1. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  2. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  3. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  4. 2023Pretraining Language Models with Human Preferencescited by 245
  5. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  6. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  7. 2024GradSafe: Detecting Jailbreak Prompts for LLMs via Safety-Critical Gradient Analysiscited by 87
  8. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 74
  9. 2023The Art of Defending: A Systematic Evaluation and Analysis of LLM Defense Strategies on Safety and Over-Defensivenesscited by 63

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 14; 2026: 29.

45 papers.

Sort:

By area:

  1. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  2. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 53
  3. 2025HiddenDetect: Detecting Jailbreak Attacks against Large Vision-Language Models via Monitoring Hidden Statescited by 29
  4. 2024Locking Down the Finetuned LLMs Safetycited by 22
  5. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  6. 2025Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behaviorcited by 12
  7. 2025Speech-Audio Compositional Attacks on Multimodal LLMs and Their Mitigation with SALMONN-Guardcited by 8
  8. 2025Training-Free Policy Violation Detection via Activation-Space Whitening in LLMscited by 6
  9. 2025C2PO: Diagnosing and Disentangling Bias Shortcuts in LLMscited by 5
  10. 2026Steering at the Source: Style Modulation Heads for Robust Persona Controlcited by 4
  11. 2025Probing Association Biases in LLM Moderation Over-Sensitivitycited by 3
  12. 2025NeuronTune: Fine-Grained Neuron Modulation for Balanced Safety-Utility Alignment in LLMscited by 3
33 more
  1. 2025Beyond Over-Refusal: Scenario-Based Diagnostics and Post-Hoc Mitigation for Exaggerated Refusals in LLMscited by 3
  2. 2026Steering to Say No: Configurable Refusal via Activation Steering in Vision Language Modelscited by 3
  3. 2026Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive Refinementcited by 3
  4. 2026Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systemscited by 3
  5. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  6. 2026Preventing Safety Drift in Large Language Models via Coupled Weight and Activation Constraintscited by 2
  7. 2025Steering Over-refusals Towards Safety in Retrieval Augmented Generationcited by 1
  8. 2025Reimagining Safety Alignment with An Imagecited by 1
  9. 2026DriveSafe: A Hierarchical Risk Taxonomy for Safety-Critical LLM-Based Driving Assistantscited by 1
  10. 2026TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Modelscited by 1
  11. 2026From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directionscited by 1
  12. 2026Multilingual Safety Alignment via Self-Distillationcited by 1
  13. 2026Paved with True Intents: Intent-Aware Training Improves LLM Safety Classification Across Training Regimescited by 1
  14. 2025MASteer: Multi-Agent Adaptive Steer Strategy for End-to-End LLM Trustworthiness Repaircited by 0
  15. 2026LLM-VA: Resolving the Jailbreak-Overrefusal Trade-off via Vector Alignmentcited by 0
  16. 2026Asking Forever: Universal Activations Behind Turn Amplification in Conversational LLMscited by 0
  17. 2026Gradient-Controlled Decoding: A Safety Guardrail for LLMs with Dual-Anchor Steeringcited by 0
  18. 2026A Systematic Study of Training-Free Methods for Trustworthy Large Language Modelscited by 0
  19. 2026Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decodingcited by 0
  20. 2026Characterizing Model-Native Skillscited by 0
  21. 2026Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detectioncited by 0
  22. 2026Gradient-Guided Reward Optimization for Inference-time Alignmentcited by 0
  23. 2026AOR-Bench: Do Large Audio Language Models Over-Refuse Pseudo-Harmful Queries?cited by 0
  24. 2026HARC: Coupling Harmfulness and Refusal Directions for Robust Safety Alignmentcited by 0
  25. 2026Robust Critics: Defending LLMs Against Multi-Turn Attackscited by 0
  26. 2026CLEAR: Continuous Latent Adapter Routing for Utility-Preserving LLM Safety Alignmentcited by 0
  27. 2026Who Pays More for Safety? Measuring the Disparate Cost of Safety Alignment across Languagescited by 0
  28. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustnesscited by 0
  29. 2026Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Modelscited by 0
  30. 2026SafeTune: A Unified Faithful Library for Auditing and Repairing Safety Drift in Fine-Tuned LLMscited by 0
  31. 2026Mitigating LLM Over-Refusal via Dynamic Semantic Routing Calibrationcited by 0
  32. 2026Refusing Everything Looks Safe: Restoring the Benign Arm to Encoded-Prompt Evaluationcited by 0
  33. 2026Steering Fields: Adaptive Vector Fields for Safe Image Generation and Beyondcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Activation Approximations Can Incur Safety Vulnerabilities Even in Aligned LLMs: Comprehensive Analysis and Defense
  2. 2026The Geometry of Refusal: Linear Instability in Safety-Aligned LLMs
  3. 2026LLMs Can Unlearn Refusal with Only 1,000 Benign Samples
  4. 2026RAS: Measuring LLM Safety Through Refusal Alignment
  5. 2026Safety at One Shot: Patching Fine-Tuned LLMs with A Single Instance
  6. 2026Safety Paradox: How Enhanced Safety Awareness Leaves LLMs Vulnerable to Posterior Attack
  7. 2026Few Tokens, Big Leverage: Preserving Safety Alignment by Constraining Safety Tokens during Fine-tuning
  8. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning