# Enhancing Automated Interpretability with Output-Centric Feature Descriptions

arXiv: https://arxiv.org/abs/2501.08319 · HTML: https://mlsanity.com/paper/2501.08319/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 14 Jan 2025.

## Builds on (most cited first, 30 of 39)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [k-Sparse Autoencoders](https://mlsanity.com/paper/1312.5663/) (2013)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [Visualizing and Understanding Recurrent Networks](https://mlsanity.com/paper/1506.02078/) (2015)
- [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [Identifying and Controlling Important Neurons in Neural Machine Translation](https://mlsanity.com/paper/1811.01157/) (2018)
- [An Interpretability Illusion for BERT](https://mlsanity.com/paper/2104.07143/) (2021)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [Improving Steering Vectors by Targeting Sparse Autoencoder Features](https://mlsanity.com/paper/2411.02193/) (2024)
- [Neurons in Large Language Models: Dead, N-gram, Positional](https://mlsanity.com/paper/2309.04827/) (2023)
- [Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control](https://mlsanity.com/paper/2405.08366/) (2024)

## Built on by (most cited first, 30 of 41)

- [SAEs Are Good for Steering -- If You Select the Right Features](https://mlsanity.com/paper/2505.20063/) (2025)
- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [When Models Manipulate Manifolds: The Geometry of a Counting Task](https://mlsanity.com/paper/2601.04480/) (2026)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior](https://mlsanity.com/paper/2605.05115/) (2026)
- [AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery](https://mlsanity.com/paper/2604.05550/) (2026)
- [Precise In-Parameter Concept Erasure in Large Language Models](https://mlsanity.com/paper/2505.22586/) (2025)
- [Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework](https://mlsanity.com/paper/2506.15538/) (2025)
- [SAGE: An Agentic Explainer Framework for Interpreting SAE Features in Language Models](https://mlsanity.com/paper/2511.20820/) (2025)
- [Constructing Interpretable Features from Compositional Neuron Groups](https://mlsanity.com/paper/2506.10920/) (2025)
- [Evaluating SAE interpretability without explanations](https://mlsanity.com/paper/2507.08473/) (2025)
- [Circuit Insights: Towards Interpretability Beyond Activations](https://mlsanity.com/paper/2510.14936/) (2025)
- [Endogenous Resistance to Activation Steering in Language Models](https://mlsanity.com/paper/2602.06941/) (2026)
- [Evaluating Neuron Explanations: A Unified Framework with Sanity Checks](https://mlsanity.com/paper/2506.05774/) (2025)
- [Learning Distribution-Wise Control in Representation Space for Language Models](https://mlsanity.com/paper/2506.06686/) (2025)
- [Deceptive Automated Interpretability: Language Models Coordinating to Fool Oversight Systems](https://mlsanity.com/paper/2504.07831/) (2025)
- [Interpreting Language Models Through Concept Descriptions: A Survey](https://mlsanity.com/paper/2510.01048/) (2025)
- [Atlas-Alignment: Making Interpretability Transferable Across Language Models](https://mlsanity.com/paper/2510.27413/) (2025)
- [Disentangling MLP Neuron Weights in Vocabulary Space](https://mlsanity.com/paper/2604.06005/) (2026)
- [Decoding Vision Transformers: the Diffusion Steering Lens](https://mlsanity.com/paper/2504.13763/) (2025)
- [Steering Information Utility in Key-Value Memory for Language Model Post-Training](https://mlsanity.com/paper/2507.05158/) (2025)
- [Insights into a radiology-specialised multimodal large language model with sparse autoencoders](https://mlsanity.com/paper/2507.12950/) (2025)
- [Semantic Regexes: Auto-Interpreting LLM Features with a Structured Language](https://mlsanity.com/paper/2510.06378/) (2025)
- [Pitfalls in Evaluating Interpretability Agents](https://mlsanity.com/paper/2603.20101/) (2026)
- [Language Models Can Explain Visual Features via Steering](https://mlsanity.com/paper/2603.22593/) (2026)
- [Interpreto: An Explainability Library for Transformers](https://mlsanity.com/paper/2512.09730/) (2025)
- [Beyond Activation Patterns: A Weight-Based Out-of-Context Explanation of Sparse Autoencoder Features](https://mlsanity.com/paper/2601.22447/) (2026)
- [Prune, Interpret, Evaluate: A Cross-Layer Transcoder-Native Framework for Efficient Circuit Discovery via Feature Attribution](https://mlsanity.com/paper/2604.16889/) (2026)
- [SAEExplainer: Interpreting SAE Features with Activation-Guided Preference Optimization](https://mlsanity.com/paper/2606.08496/) (2026)
- [Beyond Top Activations: Efficient and Reliable Crowdsourced Evaluation of Automated Interpretability](https://mlsanity.com/paper/2506.07985/) (2025)

## Abstract (arXiv)

Automated interpretability pipelines generate natural language descriptions for the concepts represented by features in large language models (LLMs), such as plants or the first word in a sentence. These descriptions are derived using inputs that activate the feature, which may be a dimension or a direction in the model's representation space. However, identifying activating inputs is costly, and the mechanistic role of a feature in model behavior is determined both by how inputs cause a feature to activate and by how feature activation affects outputs. Using steering evaluations, we reveal that current pipelines provide descriptions that fail to capture the causal effect of the feature on outputs. To fix this, we propose efficient, output-centric methods for automatically generating feature descriptions. These methods use the tokens weighted higher after feature stimulation or the highest weight tokens after applying the vocabulary "unembedding" head directly to the feature. Our output-centric descriptions better capture the causal effect of a feature on model outputs than input-centric descriptions, but combining the two leads to the best performance on both input and output evaluations. Lastly, we show that output-centric descriptions can be used to find inputs that activate features previously thought to be "dead".