# Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers

arXiv: https://arxiv.org/abs/2501.17727 · HTML: https://mlsanity.com/paper/2501.17727/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- First posted on arXiv 29 Jan 2025.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Built on by (most cited first, 30 of 30)

- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Incorporating Hierarchical Semantics in Sparse Autoencoder Architectures](https://mlsanity.com/paper/2506.01197/) (2025)
- [FADE: Why Bad Descriptions Happen to Good Features](https://mlsanity.com/paper/2502.16994/) (2025)
- [The Dead Salmons of AI Interpretability](https://mlsanity.com/paper/2512.18792/) (2025)
- [Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations](https://mlsanity.com/paper/2502.18147/) (2025)
- [AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Features](https://mlsanity.com/paper/2510.00404/) (2025)
- [OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Features](https://mlsanity.com/paper/2509.22033/) (2025)
- [Disentangling Polysemantic Channels in Convolutional Neural Networks](https://mlsanity.com/paper/2504.12939/) (2025)
- [Interpreting Large Text-to-Image Diffusion Models with Dictionary Learning](https://mlsanity.com/paper/2505.24360/) (2025)
- [The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features](https://mlsanity.com/paper/2509.12934/) (2025)
- [Exploratory Causal Inference in SAEnce](https://mlsanity.com/paper/2510.14073/) (2025)
- [Inference-Time Decomposition of Activations (ITDA): A Scalable Approach to Interpreting Large Language Models](https://mlsanity.com/paper/2505.17769/) (2025)
- [Interpreting Language Models Through Concept Descriptions: A Survey](https://mlsanity.com/paper/2510.01048/) (2025)
- [Are Sparse Autoencoders Useful for Java Function Bug Detection?](https://mlsanity.com/paper/2505.10375/) (2025)
- [FaithfulSAE: Towards Capturing Faithful Features with Sparse Autoencoders without External Dataset Dependencies](https://mlsanity.com/paper/2506.17673/) (2025)
- [Descriptive Collision in Sparse Autoencoder Auto-Interpretability: When One Explanation Describes Many Features](https://mlsanity.com/paper/2605.12874/) (2026)
- [Tool-Call Dependency Structure is Linearly Decodable in LLM Agent Residual Streams](https://mlsanity.com/paper/2605.25310/) (2026)
- [Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders](https://mlsanity.com/paper/2606.12138/) (2026)
- [SplInterp: Improving our Understanding and Training of Sparse Autoencoders](https://mlsanity.com/paper/2505.11836/) (2025)
- [Explaining Neural Networks with Reasons](https://mlsanity.com/paper/2505.14424/) (2025)
- [Kronecker Factorization Improves Efficiency and Interpretability of Sparse Autoencoders](https://mlsanity.com/paper/2505.22255/) (2025)
- [Perspectives on a Reliability Monitoring Framework for Agentic AI Systems](https://mlsanity.com/paper/2511.09178/) (2025)
- [SMolLM: Small Language Models Learn Small Molecular Grammar](https://mlsanity.com/paper/2605.06322/) (2026)
- [Decompose Sparsely Where You Should, Absorb Densely Where You Should No](https://mlsanity.com/paper/2606.14040/) (2026)
- [Scaling Interpretable Transformers with Parity Bottleneck Layers](https://mlsanity.com/paper/2607.20652/) (2026)
- [Where You Measure Decides What You Measure: Position Selection in Ablation-Based SAE Evaluation](https://mlsanity.com/paper/2608.13337/) (2026)
- [Activation-Space Order-Swap Geometry: A Site-Asymmetry Audit](https://mlsanity.com/paper/2608.25315/) (2026)
- [SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?](https://mlsanity.com/paper/2609.09113/) (2026)
- [The Misery of Mechanistic Interpretability: A Formal Perspective](https://mlsanity.com/paper/2609.15533/) (2026)

## Abstract (arXiv)

Sparse autoencoders (SAEs) are widely used to extract sparse, interpretable latents from transformer activations. We test whether commonly used SAE quality metrics and automatic explanation pipelines can distinguish trained transformers from randomly initialized ones (e.g., where parameters are sampled i.i.d. from a Gaussian). Over a wide range of Pythia model sizes and multiple randomization schemes, we find that, in many settings, SAEs trained on randomly initialized transformers produce auto-interpretability scores and reconstruction metrics that are similar to those from trained models. These results show that high aggregate auto-interpretability scores do not, by themselves, guarantee that learned, computationally relevant features have been recovered. We therefore recommend treating common SAE metrics as useful but insufficient proxies for mechanistic interpretability and argue for routine randomized baselines and targeted measures of feature 'abstractness'.