Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders
On the map: Mechanistic interpretability → SAE features and their applications
Abstract (arXiv)
Recent work in Mechanistic Interpretability (MI) has enabled the identification and intervention of internal features in Large Language Models (LLMs). However, a persistent challenge lies in linking such internal features to the reliable control of complex, behavior-level semantic attributes in language generation. In this paper, we propose a Sparse Autoencoder-based framework for retrieving and steering semantically interpretable internal features associated with high-level linguistic behaviors. Our method employs a contrastive feature retrieval pipeline based on controlled semantic oppositions, combing statistical activation analysis and generation-based validation to distill monosemantic functional features from sparse activation spaces. Using the Big Five personality traits as a case study, we demonstrate that our method enables precise, bidirectional steering of model behavior while maintaining superior stability and performance compared to existing activation steering methods like Contrastive Activation Addition (CAA). We further identify an empirical effect, which we term Functional Faithfulness, whereby intervening on a specific internal feature induces coherent and predictable shifts across multiple linguistic dimensions aligned with the target semantic attribute. Our findings suggest that LLMs internalize deeply integrated representations of high-order concepts, and provide a novel, robust mechanistic path for the regulation of complex AI behaviors.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 11 of its 42 references that are in the map. First posted on arXiv 6 Jan 2026.
Builds on
11 of 42 references.
Closest ancestors · 2
Same direction, newest first
- 2025Understanding the Repeat Curse in Large Language Models from a Feature Perspective19 Apr
- 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders27 Oct
Other references · 9
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
- 2025Persona Vectors: Monitoring and Controlling Character Traits in Language Modelscited by 345
- 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 59
- 2024BIG5-CHAT: Shaping LLM Personalities Through Training on Human-Grounded Datacited by 50
- 2023Challenging the Validity of Personality Tests for Large Language Modelscited by 25
- 2023Bias A-head? Analyzing Bias in Transformer-Based Language Model Attention Headscited by 14
- 2025LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encodercited by 14
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
- 2026Mechanistic Personality Analysis of LLMs: Steering Personality via Latent Feature Interventions
- 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders
- 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility
- 2026When the Coffee Feature Activates on Coffins: An Analysis of Feature Extraction and Steering for Mechanistic Interpretability
- 2026Sparse Auto-Encoders and Holism about Large Language Models
- 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models
- 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders