# Interpretable Embeddings with Sparse Autoencoders: A Data Analysis Toolkit

arXiv: https://arxiv.org/abs/2512.10092 · HTML: https://mlsanity.com/paper/2512.10092/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- First posted on arXiv 10 Dec 2025.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Built on by (most cited first, 11 of 11)

- [Language Model Circuits Are Sparse in the Neuron Basis](https://mlsanity.com/paper/2601.22594/) (2026)
- [Shaping capabilities with token-level data filtering](https://mlsanity.com/paper/2601.21571/) (2026)
- [Chunky Post-Training: Data Driven Failures of Generalization](https://mlsanity.com/paper/2602.05910/) (2026)
- [Automatically Finding Reward Model Biases](https://mlsanity.com/paper/2602.15222/) (2026)
- [Simple LLM Baselines are Competitive for Model Diffing](https://mlsanity.com/paper/2602.10371/) (2026)
- [Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal](https://mlsanity.com/paper/2606.12360/) (2026)
- [Data-Centric Interpretability for LLM-based Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2602.05183/) (2026)
- [In your own words: computationally identifying interpretable themes in free-text survey data](https://mlsanity.com/paper/2603.26930/) (2026)
- [Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?](https://mlsanity.com/paper/2606.22994/) (2026)
- [Analysing LLM Persona Generation and Fairness Interpretation in Polarised Geopolitical Contexts](https://mlsanity.com/paper/2603.22837/) (2026)
- [RouterInterp: Understanding Superposed Specialisation in Mixture of Experts Routing](https://mlsanity.com/paper/2610.11775/) (2026)

## Abstract (arXiv)

Analyzing large-scale text corpora is a core challenge in machine learning, crucial for tasks like identifying undesirable model behaviors or biases in training data. Current methods often rely on costly LLM-based techniques (e.g. annotating dataset differences) or dense embedding models (e.g. for clustering), which lack control over the properties of interest. We propose using sparse autoencoders (SAEs) to create SAE embeddings: representations whose dimensions map to interpretable concepts. Through four data analysis tasks, we show that SAE embeddings are more cost-effective and reliable than LLMs and more controllable than dense embeddings. Using the large hypothesis space of SAEs, we can uncover insights such as (1) semantic differences between datasets and (2) unexpected concept correlations in documents. For instance, by comparing model responses, we find that Grok-4 clarifies ambiguities more often than nine other frontier models. Relative to LLMs, SAE embeddings uncover bigger differences at 2-8x lower cost and identify biases more reliably. Additionally, SAE embeddings are controllable: by filtering concepts, we can (3) cluster documents along axes of interest and (4) outperform dense embeddings on property-based retrieval. Using SAE embeddings, we study model behavior with two case studies: investigating how OpenAI model behavior has changed over time and finding "trigger" phrases learned by Tulu-3 (Lambert et al., 2024) from its training data. These results position SAEs as a versatile tool for unstructured data analysis and highlight the neglected importance of interpreting models through their data.