mlsanityData to 14 Oct 2026

Towards Inference-time Category-wise Safety Steering for Large Language Models

arXiv 2410.01174 ↗First posted 2 Oct 2024cs.CL cs.AIcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 2 more
Abstract (arXiv)

While large language models (LLMs) have seen unprecedented advancements in capabilities and applications across a variety of use-cases, safety alignment of these models is still an area of active research. The fragile nature of LLMs, even models that have undergone extensive alignment and safety training regimes, warrants additional safety steering steps via training-free, inference-time methods. While recent work in the area of mechanistic interpretability has investigated how activations in latent representation spaces may encode concepts, and thereafter performed representation engineering to induce such concepts in LLM outputs, the applicability of such for safety is relatively under-explored. Unlike recent inference-time safety steering works, in this paper we explore safety steering of LLM outputs using: (i) category-specific steering vectors, thereby enabling fine-grained control over the steering, and (ii) sophisticated methods for extracting informative steering vectors for more effective safety steering while retaining quality of the generated text. We demonstrate our exploration on multiple LLMs and datasets, and showcase the effectiveness of the proposed steering method, along with a discussion on the implications and best practices.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 36 of its 42 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

36 of 42 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing11 Jul
  2. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity22 May
  3. 2024Language Models are Homer Simpson! Safety Re-Alignment of Fine-tuned Language Models through Task Arithmetic19 Feb
  4. 2024InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance20 Jan
  5. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity3 Jan
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  8. 2023Are aligned neural networks adversarially aligned?26 Jun

Other references · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2022Emergent Abilities of Large Language Modelscited by 2,512
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  7. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  8. 2022Editing Models with Task Arithmeticcited by 1,120
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  11. 2023Steering Language Models With Activation Engineeringcited by 922
  12. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
16 more
  1. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  2. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  3. 2024Scaling and evaluating sparse autoencoderscited by 553
  4. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  5. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  6. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 179
  7. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 157
  8. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editingcited by 79
  9. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 62
  10. 2024Spectral Editing of Activations for Large Language Model Alignmentcited by 56
  11. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  12. 2024Steering Without Side Effects: Improving Post-Deployment Control of Language Modelscited by 46
  13. 2023Towards LLM-guided Causal Explainability for Black-box Text Classifierscited by 44
  14. 2024A Language Model's Guide Through Latent Spacecited by 43
  15. 2024From Theft to Bomb-Making: The Ripple Effect of Unlearning in Defending Against Jailbreak Attackscited by 29
  16. 2024Parameter-Efficient Detoxification with Contrastive Decodingcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 6.

18 papers.

Sort:

By area:

  1. 2024What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attackscited by 29
  2. 2025To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Modelscited by 29
  3. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  4. 2025RepIt: Steering Language Models with Concept-Specific Refusal Vectorscited by 12
  5. 2026Spectral Attention Steering for Prompt Highlightingcited by 8
  6. 2025Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attackscited by 7
  7. 2026Fine-Grained Activation Steering: Steering Less, Achieving Morecited by 7
  8. 2025Disentangled Safety Adapters Enable Efficient Guardrails and Flexible Inference-Time Alignmentcited by 6
  9. 2024Focus On This, Not That! Steering LLMs with Adaptive Feature Specificationcited by 5
  10. 2025PII Jailbreaking in LLMs via Activation Steering Reveals Personal Information Leakagecited by 5
  11. 2025Turning the Spell Around: Lightweight Alignment Amplification via Rank-One Safety Injectioncited by 5
  12. 2025ExpertSteer: Intervening in LLMs through Expert Knowledgecited by 4
6 more
  1. 2025A Mixture of Linear Corrections Generates Secure Codecited by 4
  2. 2026Why Do LLMs Struggle in Strategic Play? Broken Links Between Observations, Beliefs, and Actionscited by 3
  3. 2025Annotating the Chain-of-Thought: A Behavior-Labeled Dataset for AI Safetycited by 1
  4. 2026DSPA: Dynamic SAE Steering for Data-Efficient Preference Alignmentcited by 1
  5. 2026A Systematic Study of Training-Free Methods for Trustworthy Large Language Modelscited by 0
  6. 2026How to Tame a Multi-Headed Hydra? Adaptive Multi-Category Safety Steering for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SafeSteer: Interpretable Safety Steering with Refusal-Evasion in LLMs
  2. 2026Probabilistic Concept-Aware Steering for Trustworthy LLM Inference
  3. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
  4. 2026Steer2Adapt: Dynamically Composing Steering Vectors Elicits Efficient Adaptation of LLMs
  5. 2024Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models
  6. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
  7. 2026LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion
  8. 2026FineSteer: A Unified Framework for Fine-Grained Inference-Time Steering in Large Language Models