From Geometric Recovery to Causal Validation: A Reproducible Audit of Sparse Autoencoder Features, from Superposition Geometry to Causal Inertness
Not on the map. Why?
Abstract (arXiv)
Sparse autoencoders (SAEs) are the standard for decomposing superposed neural representations into interpretable features, and evaluation relies predominantly on correlational recovery metrics -- cosine similarity between ground-truth directions and decoder atoms. We show this conflates two distinct claims: decoder-geometry alignment and encoder-activation behavior. We reproduce the superposition phase diagram of Elhage et al. (2022), identifying a convergence artifact at high sparsity and an under-described diffuse sharing regime at extreme overcompleteness. We reproduce the TopK-versus-L1 comparison of Gao et al. (2024), with direct evidence of L1 shrinkage. Our central result is causal: subjecting every recovered feature to ablation and steering, we find up to 77% of features passing a recovery bar (cosine >= 0.90) in a degraded SAE -- and 9% in a well-trained one -- are causally inert: the matched atom never fires when the feature is present, including matches at cosine ~1.000. We package the method as sae-causal-audit, a model-agnostic instrument with a deterministic pipeline. Re-auditing refines the finding: inertness decomposes by cause into structural inertness (antipodal-pair geometry, present in good SAEs) and competitive inertness (a TopK pathology of degraded SAEs), and by direction into read- and write-inertness, which five antipodal pairs dissociate completely -- unmonitorable yet steerable through the same atom, with steering specificities of 143-310 attached to zero ablation effects. We document why byte-exact reproducibility is unavailable by construction, and propose reporting it as a stack of claims with explicit scopes. Applying the instrument to a production SAE reproduces the pattern at small scale (14% inert) and surfaces an atom-collision signal: a handful of atoms recur as the nearest match for dozens of unrelated concepts, replicated across three batches.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 9 of its 15 references that are in the map. First posted on arXiv 13 Jul 2026.
Builds on
9 of 15 references.
References · 9
- 2022Toy Models of Superpositioncited by 928
- 2024Scaling and evaluating sparse autoencoderscited by 530
- 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
- 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
- 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 175
- 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
- 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 148
- 2024BatchTopK Sparse Autoencoderscited by 126
- 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 118
Built on by
Not cited yet.