# Interpretable Steering of Large Language Models with Feature Guided Activation Additions

arXiv: https://arxiv.org/abs/2501.09929 · HTML: https://mlsanity.com/paper/2501.09929/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 16 of its 24 references that are in the map.
- First posted on arXiv 17 Jan 2025.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 16 of 16)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024)
- [Automatically Interpreting Millions of Features in Large Language Models](https://mlsanity.com/paper/2410.13928/) (2024)
- [NEFTune: Noisy Embeddings Improve Instruction Finetuning](https://mlsanity.com/paper/2310.05914/) (2023)
- [Improving Steering Vectors by Targeting Sparse Autoencoder Features](https://mlsanity.com/paper/2411.02193/) (2024)
- [Polysemanticity and Capacity in Neural Networks](https://mlsanity.com/paper/2210.01892/) (2022)

## Built on by (most cited first, 30 of 35)

- [I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2503.18878/) (2025)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [The Rogue Scalpel: Activation Steering Compromises LLM Safety](https://mlsanity.com/paper/2509.22067/) (2025)
- [Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models](https://mlsanity.com/paper/2603.19233/) (2026)
- [Causal Language Control in Multilingual Transformers via Sparse Feature Steering](https://mlsanity.com/paper/2507.13410/) (2025)
- [CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features](https://mlsanity.com/paper/2508.12535/) (2025)
- [Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety](https://mlsanity.com/paper/2506.05451/) (2025)
- [Enhancing LLM Steering through Sparse Autoencoder-Based Vector Refinement](https://mlsanity.com/paper/2509.23799/) (2025)
- [The Geometry of Harmfulness in LLMs through Subconcept Probing](https://mlsanity.com/paper/2507.21141/) (2025)
- [What Can We Actually Steer? A Multi-Behavior Study of Activation Control](https://mlsanity.com/paper/2511.18284/) (2025)
- [Steering at the Source: Style Modulation Heads for Robust Persona Control](https://mlsanity.com/paper/2603.13249/) (2026)
- [Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoders](https://mlsanity.com/paper/2505.08080/) (2025)
- [Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selection](https://mlsanity.com/paper/2605.23036/) (2026)
- [Bottlenecked Transformers: Periodic KV Cache Consolidation for Generalised Reasoning](https://mlsanity.com/paper/2505.16950/) (2025)
- [Abstract Activation Spaces for Content-Invariant Reasoning in Large Language Models](https://mlsanity.com/paper/2602.02462/) (2026)
- [Deployable Per-Instance Multi-Layer Activation Steering for Large Language Models](https://mlsanity.com/paper/2608.08829/) (2026)
- [Steering Information Utility in Key-Value Memory for Language Model Post-Training](https://mlsanity.com/paper/2507.05158/) (2025)
- [Painless Activation Steering: An Automated, Lightweight Approach for Post-Training Large Language Models](https://mlsanity.com/paper/2509.22739/) (2025)
- [Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis](https://mlsanity.com/paper/2603.17624/) (2026)
- [Fusion Steering: Prompt-Specific Activation Control](https://mlsanity.com/paper/2505.22572/) (2025)
- [Steerability of Instrumental-Convergence Tendencies in LLMs](https://mlsanity.com/paper/2601.01584/) (2026)
- [Prometheus Mind: Retrofitting Memory to Frozen Language Models](https://mlsanity.com/paper/2601.15324/) (2026)
- [Why Are Some Emotions Harder for LLMs? Uncovering the Causal Mechanisms of Emotion Inference via Sparse Autoencoders](https://mlsanity.com/paper/2604.25866/) (2026)
- [REALISTA: Realistic Latent Adversarial Attacks that Elicit LLM Hallucinations](https://mlsanity.com/paper/2605.12813/) (2026)
- [Mechanistic Interpretability for Neural Networks: Circuits, Sparse Features and Symbolic Reasoning](https://mlsanity.com/paper/2607.07316/) (2026)
- [Beyond topography: Topographic regularization improves robustness and reshapes representations in convolutional neural networks](https://mlsanity.com/paper/2508.00043/) (2025)
- [Mechanistic Indicators of Steering Effectiveness in Large Language Models](https://mlsanity.com/paper/2602.01716/) (2026)
- [Continuous Interpretive Steering for Scalar Diversity](https://mlsanity.com/paper/2604.07006/) (2026)
- [Transcoders Trace Visual Grounding and Hallucinations in Vision-Language Models](https://mlsanity.com/paper/2605.22902/) (2026)
- [Steered Generation via Gradient-Based Optimization on Sparse Query Features](https://mlsanity.com/paper/2605.23040/) (2026)

## Abstract (arXiv)

Effective and reliable control over large language model (LLM) behavior is a significant challenge. While activation steering methods, which add steering vectors to a model's hidden states, are a promising approach, existing techniques often lack precision and interpretability in how they influence model outputs. We introduce Feature Guided Activation Additions (FGAA), a novel activation steering method that leverages insights from Contrastive Activation Addition (CAA) and Sparse Autoencoder-Targeted Steering (SAE-TS). By operating in the latent space of a Sparse Autoencoder (SAE) and employing optimization techniques to select desired SAE features, FGAA constructs precise steering vectors that provide better steering effects while maintaining coherence of steered model outputs. In this regard, evaluations on Gemma-2-2B and Gemma-2-9B models across various steering tasks demonstrate that FGAA outperforms existing steering methods of CAA, SAE decoder steering, and SAE-TS. Our results also highlight important trade-offs between steering scale and general model capabilities that are consistent across all tested steering methods.