mlsanityData to 7 Oct 2026

Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Small

arXiv 2409.04478 ↗First posted 5 Sep 2024cs.LG cs.AI cs.NEcited by 34

Not on the map. Why?

Abstract (arXiv)

A popular new method in mechanistic interpretability is to train high-dimensional sparse autoencoders (SAEs) on neuron activations and use SAE features as the atomic units of analysis. However, the body of evidence on whether SAE feature spaces are useful for causal analysis is underdeveloped. In this work, we use the RAVEL benchmark to evaluate whether SAEs trained on hidden representations of GPT-2 small have sets of features that separately mediate knowledge of which country a city is in and which continent it is in. We evaluate four open-source SAEs for GPT-2 small against each other, with neurons serving as a baseline, and linear features learned via distributed alignment search (DAS) serving as a skyline. For each, we learn a binary mask to select features that will be patched to change the country of a city without changing the continent, or vice versa. Our results show that SAEs struggle to reach the neuron baseline, and none come close to the DAS skyline. We release code here: https://github.com/MaheepChaudhary/SAE-Ravel

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 23 of its 39 references that are in the map. First posted on arXiv 5 Sep 2024.

Builds on

23 of 39 references.

References · 23

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2022Mass-Editing Memory in a Transformercited by 976
  7. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  8. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  9. 2021Causal Abstractions of Neural Networkscited by 401
  10. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  11. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  12. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
11 more
  1. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  2. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  3. 2022Natural Language Descriptions of Deep Visual Featurescited by 136
  4. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  5. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  6. 2021An Interpretability Illusion for BERTcited by 94
  7. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  8. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 64
  9. 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
  10. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  11. 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 21; 2026: 9.

34 papers.

Sort:

By area:

  1. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 118
  2. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  3. 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
  4. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  5. 2025LLM Pretraining with Continuous Conceptscited by 33
  6. 2025Steering off Course: Reliability Challenges in Steering Language Modelscited by 29
  7. 2025Beyond Prompt Engineering: Robust Behavior Control in LLMs via Steering Target Atomscited by 23
  8. 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoderscited by 21
  9. 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
  10. 2025Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Wordscited by 17
  11. 2024Analyzing (In)Abilities of SAEs via Formal Languagescited by 16
  12. 2025SALT: Steering Activations towards Leakage-free Thinking in Chain of Thoughtcited by 11
22 more
  1. 2026From Directions to Regions: Decomposing Activations in Language Models via Local Geometrycited by 10
  2. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Featurescited by 8
  3. 2026Causality is Key for Interpretability Claims to Generalisecited by 8
  4. 2025Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonalitycited by 6
  5. 2025Self-Regularization with Sparse Autoencoders for Controllable LLM-based Classificationcited by 4
  6. 2025LLM Interpretability with Identifiable Temporal-Instantaneous Representationcited by 4
  7. 2025Sparse Autoencoders, Again?cited by 3
  8. 2025Constructing Interpretable Features from Compositional Neuron Groupscited by 3
  9. 2025A Comparative Analysis of Sparse Autoencoder and Activation Difference in Language Model Steeringcited by 3
  10. 2025Are Sparse Autoencoders Useful for Java Function Bug Detection?cited by 2
  11. 2026Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?cited by 2
  12. 2025Automated Neuron Labelling Enables Generative Steering and Interpretability in Protein Language Modelscited by 1
  13. 2025ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Modelscited by 1
  14. 2026Why Limit the Residual Stream to Layers and Not Tokens? Persistent Memory for Continuous Latent Reasoningcited by 1
  15. 2026Decodability is Not Causality: Dissociating Probe Readouts from Behavioral Drivers via SAE Decompositioncited by 1
  16. 2024Deep Model Merging: The Sister of Neural Network Interpretability -- A Surveycited by 0
  17. 2025Discovering Chunks in Neural Embeddings for Interpretabilitycited by 0
  18. 2025Concept-Guided Interpretability via Neural Chunkingcited by 0
  19. 2026PLOT: Progressive Localization via Optimal Transport in Neural Causal Abstractioncited by 0
  20. 2026SAERec: Constructing Fine-grained Interpretable Intents Priors via Sparse Autoencoders for Recommendationcited by 0
  21. 2026Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effectscited by 0
  22. 2026Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluationcited by 0