# Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

arXiv: https://arxiv.org/abs/2409.04478 · HTML: https://mlsanity.com/paper/2409.04478/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 5 Sep 2024.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 23 of 23)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models](https://mlsanity.com/paper/2106.06087/) (2021)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [An Interpretability Illusion for BERT](https://mlsanity.com/paper/2104.07143/) (2021)
- [Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control](https://mlsanity.com/paper/2405.08366/) (2024)
- [Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning](https://mlsanity.com/paper/2405.12241/) (2024)
- [How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking](https://mlsanity.com/paper/2004.14992/) (2020)
- [Sparse Interventions in Language Models with Differentiable Masking](https://mlsanity.com/paper/2112.06837/) (2021)
- [FIND: A Function Description Benchmark for Evaluating Interpretability Methods](https://mlsanity.com/paper/2309.03886/) (2023)

## Built on by (most cited first, 30 of 34)

- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Steering Language Model Refusal with Sparse Autoencoders](https://mlsanity.com/paper/2411.11296/) (2024)
- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [LLM Pretraining with Continuous Concepts](https://mlsanity.com/paper/2502.08524/) (2025)
- [Steering off Course: Reliability Challenges in Steering Language Models](https://mlsanity.com/paper/2504.04635/) (2025)
- [Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atoms](https://mlsanity.com/paper/2505.20322/) (2025)
- [Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders](https://mlsanity.com/paper/2502.15576/) (2025)
- [Residual Stream Analysis with Multi-Layer SAEs](https://mlsanity.com/paper/2409.04185/) (2024)
- [Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words](https://mlsanity.com/paper/2501.06254/) (2025)
- [Analyzing (In)Abilities of SAEs via Formal Languages](https://mlsanity.com/paper/2410.11767/) (2024)
- [From Directions to Regions: Decomposing Activations in Language Models via Local Geometry](https://mlsanity.com/paper/2602.02464/) (2026)
- [SALT: Steering Activations towards Leakage-free Thinking in Chain of Thought](https://mlsanity.com/paper/2511.07772/) (2025)
- [AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features](https://mlsanity.com/paper/2510.00404/) (2025)
- [Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality](https://mlsanity.com/paper/2503.24277/) (2025)
- [Causality is Key for Interpretability Claims to Generalise](https://mlsanity.com/paper/2602.16698/) (2026)
- [LLM Interpretability with Identifiable Temporal-Instantaneous Representation](https://mlsanity.com/paper/2509.23323/) (2025)
- [Self-Regularization with Sparse Autoencoders for Controllable LLM-based Classification](https://mlsanity.com/paper/2502.14133/) (2025)
- [Constructing Interpretable Features from Compositional Neuron Groups](https://mlsanity.com/paper/2506.10920/) (2025)
- [A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steering](https://mlsanity.com/paper/2510.01246/) (2025)
- [Sparse Autoencoders, Again?](https://mlsanity.com/paper/2506.04859/) (2025)
- [Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?](https://mlsanity.com/paper/2605.28860/) (2026)
- [Are Sparse Autoencoders Useful for Java Function Bug Detection?](https://mlsanity.com/paper/2505.10375/) (2025)
- [Decodability is Not Causality: Dissociating Probe Readouts from Behavioral Drivers via SAE Decomposition](https://mlsanity.com/paper/2609.18080/) (2026)
- [Concept-Guided Interpretability via Neural Chunking](https://mlsanity.com/paper/2505.11576/) (2025)
- [Automated Neuron Labelling Enables Generative Steering and Interpretability in Protein Language Models](https://mlsanity.com/paper/2507.06458/) (2025)
- [ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Models](https://mlsanity.com/paper/2509.20376/) (2025)
- [Why Limit the Residual Stream to Layers and Not Tokens? Persistent Memory for Continuous Latent Reasoning](https://mlsanity.com/paper/2606.07720/) (2026)
- [Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects](https://mlsanity.com/paper/2607.24645/) (2026)
- [Deep Model Merging: The Sister of Neural Network Interpretability -- A Survey](https://mlsanity.com/paper/2410.12927/) (2024)

## Abstract (arXiv)

A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline, and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline. We release code here: https://github.com/MaheepChaudhary/SAE-Ravel