# Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

arXiv: https://arxiv.org/abs/2605.29358 · HTML: https://mlsanity.com/paper/2605.29358/

## Key facts

- Cited by 191 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 33 of its 77 references that are in the map.
- First posted on arXiv 28 May 2026.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 30 of 33)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training](https://mlsanity.com/paper/2401.05566/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [On the "steerability" of generative adversarial networks](https://mlsanity.com/paper/1907.07171/) (2019)
- [Attribution Patching Outperforms Automated Circuit Discovery](https://mlsanity.com/paper/2310.10348/) (2023)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [AtP*: An efficient and scalable method for localizing LLM behaviour to components](https://mlsanity.com/paper/2403.00745/) (2024)
- [Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factors](https://mlsanity.com/paper/2103.15949/) (2021)
- [Polysemanticity and Capacity in Neural Networks](https://mlsanity.com/paper/2210.01892/) (2022)
- [Eliciting Latent Knowledge from Quirky Language Models](https://mlsanity.com/paper/2312.01037/) (2023)
- [Codebook Features: Sparse and Discrete Interpretability for Neural Networks](https://mlsanity.com/paper/2310.17230/) (2023)

## Built on by (most cited first, 30 of 191)

- [Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden States](https://mlsanity.com/paper/2608.02657/) (2026)
- [Perplexity Can Miss SAE Feature Damage Under Quantization](https://mlsanity.com/paper/2606.03002/) (2026)
- [ICA Lens: Interpreting Language Models Without Training Another Dictionary](https://mlsanity.com/paper/2606.11722/) (2026)
- [Beyond representational alignment with brain-guided language models for robust reasoning](https://mlsanity.com/paper/2606.11893/) (2026)
- [HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridization](https://mlsanity.com/paper/2606.20097/) (2026)
- [Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines](https://mlsanity.com/paper/2605.31183/) (2026)
- [Beyond Importance: Interchange-Sobol Sensitivity Reveals Task-Specific Content Channels in Transformer Components](https://mlsanity.com/paper/2606.20678/) (2026)
- [Mechanistically Eliciting Latent Behaviors in Language Models](https://mlsanity.com/paper/2606.29604/) (2026)
- [PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis](https://mlsanity.com/paper/2607.12624/) (2026)
- [Decoding Task Progress from VLA Representations](https://mlsanity.com/paper/2608.13474/) (2026)
- [LLM Layers Immediately Correct Each Other](https://mlsanity.com/paper/2609.07876/) (2026)
- [On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoders](https://mlsanity.com/paper/2605.31518/) (2026)
- [The Case for Model Science: Verify, Explore, Steer, Refine](https://mlsanity.com/paper/2606.01189/) (2026)
- [Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Models](https://mlsanity.com/paper/2606.05378/) (2026)
- [Knowledge-Based Zero-Replay Debugging of Multi-Agent LLM Traces](https://mlsanity.com/paper/2606.14805/) (2026)
- [What Does a Chemical Language Model Know About Molecules?](https://mlsanity.com/paper/2606.23443/) (2026)
- [Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations](https://mlsanity.com/paper/2606.24716/) (2026)
- [Interpreting "Interpretability" and Explaining "Explainability" in Machine Learning in Physics](https://mlsanity.com/paper/2606.26228/) (2026)
- [From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learning](https://mlsanity.com/paper/2606.26629/) (2026)
- [Internal-State Probes Read the Situation, Not the Action: Three Negative Results for Pre-Action Misalignment Monitoring](https://mlsanity.com/paper/2606.30449/) (2026)
- [SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Models](https://mlsanity.com/paper/2606.32022/) (2026)
- [Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillation](https://mlsanity.com/paper/2607.04432/) (2026)
- [Hidden APIs in Language Models: Discovering Reusable Causal Interfaces from Forked Futures](https://mlsanity.com/paper/2607.27617/) (2026)
- [When Is a Steerable Concept Representation Real? Measurement Confounds in a Cross-Family Audit of Neuroscience Parallels in LLMs](https://mlsanity.com/paper/2608.08159/) (2026)
- [CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Models](https://mlsanity.com/paper/2606.01695/) (2026)
- [Auditing Asset-Specific Preferences in Financial Large Language Models: Evidence from Bitcoin Representations and Portfolio Allocation](https://mlsanity.com/paper/2606.02528/) (2026)
- [Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models](https://mlsanity.com/paper/2606.02765/) (2026)
- [A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoders](https://mlsanity.com/paper/2606.07007/) (2026)
- [Beyond Accuracy: Interpreting Topic Representation in Suicide Ideation Detection Models](https://mlsanity.com/paper/2606.07714/) (2026)
- [VFUSE: Virulent Feature Understanding with Sparse autoEncoders](https://mlsanity.com/paper/2606.10080/) (2026)

## Abstract (arXiv)

We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model, addressing the open question of whether dictionary learning methods scale beyond small transformers. We trained sparse autoencoders with up to 34 million features on the model's middle layer residual stream, using scaling laws to guide hyperparameter selection. The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations. We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code. We also identify features relevant to ways in which language models might cause harm--including features representing deception, power-seeking, sycophancy, and bias--and show that these causally influence model outputs when manipulated. Additionally, we conduct analyses of feature interpretability, geometry, and computational function. However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.