# Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness

arXiv: https://arxiv.org/abs/2410.21331 · HTML: https://mlsanity.com/paper/2410.21331/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 61 references that are in the map.
- First posted on arXiv 27 Oct 2024.

## Builds on (most cited first, 30 of 43)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Learning Robust Global Representations by Penalizing Local Predictive Power](https://mlsanity.com/paper/1905.13549/) (2019)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Learning from Noisy Labels with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2007.08199/) (2020)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [Large Language Models: A Survey](https://mlsanity.com/paper/2402.06196/) (2024)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [ShieldGemma: Generative AI Content Moderation Based on Gemma](https://mlsanity.com/paper/2407.21772/) (2024)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [When Scaling Meets LLM Finetuning: The Effect of Data, Model and Finetuning Method](https://mlsanity.com/paper/2402.17193/) (2024)
- [Dimensionality-Driven Learning with Noisy Labels](https://mlsanity.com/paper/1806.02612/) (2018)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)

## Built on by (most cited first, 13 of 13)

- [On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedy](https://mlsanity.com/paper/2506.15963/) (2025)
- [Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models](https://mlsanity.com/paper/2503.06269/) (2025)
- [Visual Self-Fulfilling Alignment: Shaping Safety-Oriented Personas via Threat-Related Images](https://mlsanity.com/paper/2603.08486/) (2026)
- [SAE as a Crystal Ball: Interpretable Features Predict Cross-domain Transferability of LLMs without Training](https://mlsanity.com/paper/2603.02908/) (2026)
- [Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms](https://mlsanity.com/paper/2604.00012/) (2026)
- [Extracting Interaction-Aware Monosemantic Concepts in Recommender Systems](https://mlsanity.com/paper/2511.18024/) (2025)
- [SoftSAE: Dynamic Top-K Selection for Adaptive Sparse Autoencoders](https://mlsanity.com/paper/2605.06610/) (2026)
- [Beyond Accuracy: Interpreting Topic Representation in Suicide Ideation Detection Models](https://mlsanity.com/paper/2606.07714/) (2026)
- [ToxScreen: Detecting Whether an LLM Has Been Poisoned](https://mlsanity.com/paper/2607.26849/) (2026)
- [Hiding in Plain Text: Detecting Concealed Jailbreaks via Activation Disentanglement](https://mlsanity.com/paper/2602.19396/) (2026)
- [EXPOSE: Explainable and Domain-Robust Embeddings from Pathology Vision Foundation Models using Sparse Autoencoders](https://mlsanity.com/paper/2608.28191/) (2026)
- [A Theoretical Framework for Masked Pretraining (MPT)](https://mlsanity.com/paper/2609.06460/) (2026)
- [Representational Simplicity and Circuit Size Dissociate in a Threshold-Dependent Way: A Controlled Test via Adversarial Training](https://mlsanity.com/paper/2609.35890/) (2026)

## Abstract (arXiv)

Deep learning models often suffer from a lack of interpretability due to polysemanticity, where individual neurons are activated by multiple unrelated semantics, resulting in unclear attributions of model behavior. Recent advances in monosemanticity, where neurons correspond to consistent and distinct semantics, have significantly improved interpretability but are commonly believed to compromise accuracy. In this work, we challenge the prevailing belief of the accuracy-interpretability tradeoff, showing that monosemantic features not only enhance interpretability but also bring concrete gains in model performance. Across multiple robust learning scenarios-including input and label noise, few-shot learning, and out-of-domain generalization-our results show that models leveraging monosemantic features significantly outperform those relying on polysemantic features. Furthermore, we provide empirical and theoretical understandings on the robustness gains of feature monosemanticity. Our preliminary analysis suggests that monosemanticity, by promoting better separation of feature representations, leads to more robust decision boundaries. This diverse evidence highlights the generality of monosemanticity in improving model robustness. As a first step in this new direction, we embark on exploring the learning benefits of monosemanticity beyond interpretability, supporting the long-standing hypothesis of linking interpretability and robustness. Code is available at urlhttps://github.com/PKU-ML/BeyondInterpretability.