# Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation

arXiv: https://arxiv.org/abs/2603.28744 · HTML: https://mlsanity.com/paper/2603.28744/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Sparse autoencoder training and evaluation, one of 13 directions in Mechanistic interpretability.
- Builds on 37 of its 68 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Placed in

- Mechanistic interpretability / [Sparse autoencoder training and evaluation](https://mlsanity.com/mechinterp/sparse-autoencoder-training-and-evaluation/)

## Builds on (most cited first, 30 of 37)

- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Unsupervised Feature Extraction by Time-Contrastive Learning and Nonlinear ICA](https://mlsanity.com/paper/1605.06336/) (2016)
- [Compositionality decomposed: how do neural networks generalise?](https://mlsanity.com/paper/1908.08351/) (2019)
- [Inference Suboptimality in Variational Autoencoders](https://mlsanity.com/paper/1801.03558/) (2018)
- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [On Linear Identifiability of Learned Representations](https://mlsanity.com/paper/2007.00810/) (2020)
- [Sparse Autoencoders Trained on the Same Data Learn Different Features](https://mlsanity.com/paper/2501.16615/) (2025)
- [Improving Steering Vectors by Targeting Sparse Autoencoder Features](https://mlsanity.com/paper/2411.02193/) (2024)
- [On the Origins of Linear Representations in Large Language Models](https://mlsanity.com/paper/2403.03867/) (2024)
- [Visual Representation Learning Does Not Generalize Strongly Within the Same Domain](https://mlsanity.com/paper/2107.08221/) (2021)
- [Additive Decoders for Latent Variables Identification and Cartesian-Product Extrapolation](https://mlsanity.com/paper/2307.02598/) (2023)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [Cross-Entropy Is All You Need To Invert the Data Generating Process](https://mlsanity.com/paper/2410.21869/) (2024)
- [Amortized Variational Inference: When and Why?](https://mlsanity.com/paper/2307.11018/) (2023)
- [Compute Optimal Inference and Provable Amortisation Gap in Sparse Autoencoders](https://mlsanity.com/paper/2411.13117/) (2024)
- [From superposition to sparse codes: interpretable representations in neural networks](https://mlsanity.com/paper/2503.01824/) (2025)
- [Feature Hedging: Correlated Features Break Narrow Sparse Autoencoders](https://mlsanity.com/paper/2505.11756/) (2025)
- [Learning biologically relevant features in a pathology foundation model using sparse autoencoders](https://mlsanity.com/paper/2407.10785/) (2024)
- [All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling](https://mlsanity.com/paper/2410.23501/) (2024)
- [Compositional Risk Minimization](https://mlsanity.com/paper/2410.06303/) (2024)
- [How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?](https://mlsanity.com/paper/2602.11246/) (2026)
- [Sparse and spurious: dictionary learning with noise and outliers](https://mlsanity.com/paper/1407.5155/) (2014)

## Built on by (most cited first, 6 of 6)

- [When Does LeJEPA Learn a World Model?](https://mlsanity.com/paper/2605.26379/) (2026)
- [The Scaling Properties of Implicit Deductive Reasoning in Transformers](https://mlsanity.com/paper/2605.04330/) (2026)
- [High-Dimensional Random Projection for Activation Steering in Language Models](https://mlsanity.com/paper/2606.15092/) (2026)
- [Adversarial Concept Search: Predicting Compositional Errors From Feature Geometry](https://mlsanity.com/paper/2606.13934/) (2026)
- [High-probability guarantees for linear accessibility in feature superposition](https://mlsanity.com/paper/2609.09556/) (2026)
- [Inference and learning in sparse autoencoders as natural gradient flow](https://mlsanity.com/paper/2610.07389/) (2026)

## Abstract (arXiv)

The linear representation hypothesis states that neural network activations encode high-level concepts as linear mixtures. However, under superposition, this encoding is a projection from a higher-dimensional concept space into a lower-dimensional activation space, and a linear decision boundary in the concept space need not remain linear after projection. In this setting, classical sparse coding methods with per-sample iterative inference leverage compressed sensing guarantees to recover latent factors. Sparse autoencoders (SAEs), on the other hand, amortise sparse inference into a fixed encoder, introducing a systematic gap. We show this amortisation gap persists across training set sizes, latent dimensions, and sparsity levels, causing SAEs to fail under out-of-distribution (OOD) compositional shifts. Through controlled experiments that decompose the failure, we identify dictionary learning as the limiting factor (not the inference procedure): SAE-learned dictionaries point in substantially wrong directions, and replacing the encoder with per-sample FISTA on the same dictionary does not close the gap. An oracle baseline proves the problem is solvable with a good dictionary at all scales tested. Our results, including experiments with real LLM activations (Pythia-70M, Gemma-2-2B) reframe the SAE failure as a dictionary learning challenge, not an inference problem, and point to scalable dictionary learning as the key open problem for sparse inference under superposition.