# When Can We Trust the Sparse Lens? A Certification Framework for SAE Faithfulness

arXiv: https://arxiv.org/abs/2606.18383 · HTML: https://mlsanity.com/paper/2606.18383/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 9 of its 16 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 9 of 9)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Uniform convergence may be unable to explain generalization in deep learning](https://mlsanity.com/paper/1902.04742/) (2019)
- [Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models](https://mlsanity.com/paper/2407.18158/) (2024)
- [LogitLens4LLMs: Extending Logit Lens Analysis to Modern Large Language Models](https://mlsanity.com/paper/2503.11667/) (2025)

## Abstract (arXiv)

Sparse autoencoders (SAEs) are increasingly used to study internal representations in language models (LMs), but it remains unclear when an SAE-based representation reliably preserves the underlying model's predictive behavior. We introduce a post-hoc certification framework for this purpose. At a chosen layer, we replace the model's hidden activation with its pretrained SAE reconstruction and derive a bound on the expected loss of the original frozen LM through this SAE-based proxy. The resulting certificate depends on three quantities measured on held-out data: the proxy risk, the reconstruction loss gap, and support mismatch between the calibration feature pool and new inputs. We further derive a more conservative exact-P extension whose proxy-risk concentration term is uniform over all feature pools of the same size. Empirically, the certificates become non-vacuous for GPT-2 Small, Gemma-2B, and Llama-3-8B. A detailed layerwise study of Llama-3-8B shows that later layers are easier to certify, driven mainly by improved reconstruction fidelity and weaker downstream amplification of reconstruction error. GPT-2 Small shows much weaker layer dependence, indicating that this pattern is not universal across model-SAE pairs. Overall, the framework provides a practical way to determine whether an SAE representation preserves enough of a frozen LM's predictive behavior to support certification of the original model. Code is available at: https://github.com/newcodevelop/sparse-lens-certification.