mlsanityData to 14 Oct 2026

Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers

arXiv 2501.17727 ↗First posted 29 Jan 2025cs.LGcited by 30

On the map: Mechanistic interpretability → Sparse autoencoder training and evaluation

Abstract (arXiv)

Sparse autoencoders (SAEs) are widely used to extract sparse, interpretable latents from transformer activations. We test whether commonly used SAE quality metrics and automatic explanation pipelines can distinguish trained transformers from randomly initialized ones (e.g., where parameters are sampled i.i.d. from a Gaussian). Over a wide range of Pythia model sizes and multiple randomization schemes, we find that, in many settings, SAEs trained on randomly initialized transformers produce auto-interpretability scores and reconstruction metrics that are similar to those from trained models. These results show that high aggregate auto-interpretability scores do not, by themselves, guarantee that learned, computationally relevant features have been recovered. We therefore recommend treating common SAE metrics as useful but insufficient proxies for mechanistic interpretability and argue for routine randomized baselines and targeted measures of feature 'abstractness'.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability. First posted on arXiv 29 Jan 2025.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 20; 2026: 10.

30 papers.

Sort:

By area:

  1. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 124
  2. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  3. 2025Incorporating Hierarchical Semantics in Sparse Autoencoder Architecturescited by 14
  4. 2025FADE: Why Bad Descriptions Happen to Good Featurescited by 13
  5. 2025The Dead Salmons of AI Interpretabilitycited by 12
  6. 2025Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activationscited by 10
  7. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Featurescited by 9
  8. 2025OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Featurescited by 8
  9. 2025Disentangling Polysemantic Channels in Convolutional Neural Networkscited by 6
  10. 2025Interpreting Large Text-to-Image Diffusion Models with Dictionary Learningcited by 6
  11. 2025The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Featurescited by 5
  12. 2025Exploratory Causal Inference in SAEncecited by 4
18 more
  1. 2025Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Modelscited by 3
  2. 2025Interpreting Language Models Through Concept Descriptions: A Surveycited by 3
  3. 2025Are Sparse Autoencoders Useful for Java Function Bug Detection?cited by 2
  4. 2025FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependenciescited by 2
  5. 2026Descriptive Collision in Sparse Autoencoder Auto-Interpretability: When One Explanation Describes Many Featurescited by 2
  6. 2026Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streamscited by 1
  7. 2026Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoderscited by 1
  8. 2025SplInterp: Improving our Understanding and Training of Sparse Autoencoderscited by 0
  9. 2025Explaining Neural Networks with Reasonscited by 0
  10. 2025Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoderscited by 0
  11. 2025Perspectives on a Reliability Monitoring Framework for Agentic AI Systemscited by 0
  12. 2026SMolLM: Small Language Models Learn Small Molecular Grammarcited by 0
  13. 2026Decompose Sparsely Where You Should, Absorb Densely Where You Should Nocited by 0
  14. 2026Scaling Interpretable Transformers with Parity Bottleneck Layerscited by 0
  15. 2026Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluationcited by 0
  16. 2026Activation-Space Order-Swap Geometry: A Site-Asymmetry Auditcited by 0
  17. 2026SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?cited by 0
  18. 2026The Misery of Mechanistic Interpretability: A Formal Perspectivecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating SAE interpretability without explanations
  2. 2024Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks
  3. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?
  4. 2024Interpreting Attention Layer Outputs with Sparse Autoencoders
  5. 2026Verifying the Linear Representation Hypothesis: How Interpretable Are Vision SAEs?
  6. 2025Position: Mechanistic Interpretability Should Prioritize Feature Consistency in SAEs
  7. 2026Improving Robustness In Sparse Autoencoders via Masked Regularization
  8. 2024Analyzing (In)Abilities of SAEs via Formal Languages