mlsanityData to 14 Oct 2026

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

arXiv 2605.29358 ↗First posted 28 May 2026cs.AIcited by 191

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model, addressing the open question of whether dictionary learning methods scale beyond small transformers. We trained sparse autoencoders with up to 34 million features on the model's middle layer residual stream, using scaling laws to guide hyperparameter selection. The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations. We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code. We also identify features relevant to ways in which language models might cause harm--including features representing deception, power-seeking, sycophancy, and bias--and show that these causally influence model outputs when manipulated. Additionally, we conduct analyses of feature interpretability, geometry, and computational function. However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.

Key facts. Cited by 191 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 33 of its 77 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

33 of 77 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Improving Dictionary Learning with Gated Sparse Autoencoders24 Apr
  2. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 31

  1. 2020Scaling Laws for Neural Language Modelscited by 6,586
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2023Steering Language Models With Activation Engineeringcited by 922
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  12. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
19 more
  1. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  2. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  3. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  4. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  5. 2023Language Models Represent Space and Timecited by 316
  6. 2023Function Vectors in Large Language Modelscited by 308
  7. 2023LEACE: Perfect linear concept erasure in closed formcited by 267
  8. 2019On the "steerability" of generative adversarial networkscited by 248
  9. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
  10. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  11. 2022Natural Language Descriptions of Deep Visual Featurescited by 137
  12. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  13. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 108
  14. 2022Polysemanticity and Capacity in Neural Networkscited by 77
  15. 2023Eliciting Latent Knowledge from Quirky Language Modelscited by 62
  16. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 48
  17. 2015Sparse Overcomplete Word Vector Representationscited by 44
  18. 2024Dictionary Learning Improves Patch-Free Circuit Discovery in Mechanistic Interpretability: A Case Study on Othello-GPTcited by 29
  19. 2019Word Embedding Visualization Via Dictionary Learningcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 191.

191 papers.

Sort:

By area:

  1. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 7
  2. 2026Perplexity Can Miss SAE Feature Damage Under Quantizationcited by 5
  3. 2026ICA Lens: Interpreting Language Models Without Training Another Dictionarycited by 4
  4. 2026Beyond representational alignment with brain-guided language models for robust reasoningcited by 4
  5. 2026HydraHead: From Head-Level Functional Heterogeneity to Specialized Attention Hybridizationcited by 4
  6. 2026Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselinescited by 3
  7. 2026Beyond Importance: Interchange-Sobol Sensitivity Reveals Task-Specific Content Channels in Transformer Componentscited by 3
  8. 2026Mechanistically Eliciting Latent Behaviors in Language Modelscited by 3
  9. 2026PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysiscited by 3
  10. 2026Decoding Task Progress from VLA Representationscited by 3
  11. 2026LLM Layers Immediately Correct Each Othercited by 3
  12. 2026On the Relationship Between Activation Outliers and Feature Death in Sparse Autoencoderscited by 2
38 more
  1. 2026The Case for Model Science: Verify, Explore, Steer, Refinecited by 2
  2. 2026Pattern Selectivity is Not Task-Causal Structure: A Cross-Architecture Mechanistic Study of Composed-Task Circuits in 1B-Class Language Modelscited by 2
  3. 2026Knowledge-Based Zero-Replay Debugging of Multi-Agent LLM Tracescited by 2
  4. 2026What Does a Chemical Language Model Know About Molecules?cited by 2
  5. 2026Evaluating the Interpretability of Sparse Autoencoders with Concept Annotationscited by 2
  6. 2026Interpreting "Interpretability" and Explaining "Explainability" in Machine Learning in Physicscited by 2
  7. 2026From Weights to Features: SAE-Guided Activation Regularization for LLM Continual Learningcited by 2
  8. 2026Internal-State Probes Read the Situation, Not the Action: Three Negative Results for Pre-Action Misalignment Monitoringcited by 2
  9. 2026SemRF: A Semantic Reference Frame for Residual-Stream Dynamics in Language Modelscited by 2
  10. 2026Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillationcited by 2
  11. 2026Hidden APIs in Language Models: Discovering Reusable Causal Interfaces from Forked Futurescited by 2
  12. 2026When Is a Steerable Concept Representation Real? Measurement Confounds in a Cross-Family Audit of Neuroscience Parallels in LLMscited by 2
  13. 2026CANARY: Zero-Label Detection of Fine-Tuning Contamination in Language Modelscited by 1
  14. 2026Auditing Asset-Specific Preferences in Financial Large Language Models: Evidence from Bitcoin Representations and Portfolio Allocationcited by 1
  15. 2026Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Modelscited by 1
  16. 2026A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoderscited by 1
  17. 2026Beyond Accuracy: Interpreting Topic Representation in Suicide Ideation Detection Modelscited by 1
  18. 2026VFUSE: Virulent Feature Understanding with Sparse autoEncoderscited by 1
  19. 2026High-Dimensional Random Projection for Activation Steering in Language Modelscited by 1
  20. 2026Constitutional Value Potentials: reading and steering internal priority margins in language modelscited by 1
  21. 2026Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMscited by 1
  22. 2026SAE Interventions are Unreliable: Post-Intervention Recovery of Suppressed Behaviorcited by 1
  23. 2026Evidence for feature-specific error correction in LLMscited by 1
  24. 2026Localizing RL-Induced Tool Use to a Single Crosscoder Featurecited by 1
  25. 2026Turn-Averaged SAEs for Feature Discovery and Long-Context Attributioncited by 1
  26. 2026Conversable Complexity: Agentic LLM Collectives as Interpretable Substratescited by 1
  27. 2026Mechanistic Interpretability and Causal Feature Steering of Neural Quantum States via Sparse Autoencoderscited by 1
  28. 2026Not All Refusals Are Equal: How Safety Alignment Fails Cybersecurity at Scalecited by 1
  29. 2026Present but Not Remembered: Auditing How Frozen VLAs Encode, Deploy, and Steer Visual Historycited by 1
  30. 2026When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systemscited by 1
  31. 2026Training, Reading, and Editing Legible Transformerscited by 1
  32. 2026The Refusal Residue: When Probes Catch Alignment Faking and When They Don'tcited by 1
  33. 2026Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agentcited by 1
  34. 2026Estimating Rare Events in Language Models with Proper Evaluationcited by 1
  35. 2026Reading and Steering Representations of Materials-Science Mechanisms in an Open-Weight Language Modelcited by 1
  36. 2026Reference Feature Atlases for Mechanistic Auditing of Language Modelscited by 1
  37. 2026How LLM Task-Adaptation Reshapes Alignment: A Multi-dimensional Study of Behavioral and Representational Driftcited by 1
  38. 2026Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effectscited by 1

50 most cited of 191 All 191 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Scaling and evaluating sparse autoencoders
  2. 2026Sparse Autoencoders for Mechanistic Interpretability in NLP: A Critical Survey of Recoverability, Interpretability, and Utility
  3. 2025Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability
  4. 2025Measuring and Guiding Monosemanticity
  5. 2026Descriptive Collision in Sparse Autoencoder Auto-Interpretability: When One Explanation Describes Many Features
  6. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  7. 2025Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words
  8. 2025Sparse Autoencoders Can Capture Language-Specific Concepts Across Diverse Languages