# Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders

arXiv: https://arxiv.org/abs/2505.16004 · HTML: https://mlsanity.com/paper/2505.16004/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 24 of its 32 references that are in the map.
- First posted on arXiv 21 May 2025.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 24 of 24)

- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [Certifying LLM Safety against Adversarial Prompting](https://mlsanity.com/paper/2309.02705/) (2023)
- [Transcoders Find Interpretable LLM Feature Circuits](https://mlsanity.com/paper/2406.11944/) (2024)
- [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024)
- [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024)
- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLP](https://mlsanity.com/paper/2210.10683/) (2022)
- [Improving Steering Vectors by Targeting Sparse Autoencoder Features](https://mlsanity.com/paper/2411.02193/) (2024)
- [Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control](https://mlsanity.com/paper/2405.08366/) (2024)
- [RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations](https://mlsanity.com/paper/2402.17700/) (2024)
- [Post-Hoc Explanations Fail to Achieve their Purpose in Adversarial Contexts](https://mlsanity.com/paper/2201.10295/) (2022)
- [Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models](https://mlsanity.com/paper/2408.00113/) (2024)
- [Towards Unifying Interpretability and Control: Evaluation via Intervention](https://mlsanity.com/paper/2411.04430/) (2024)
- [Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders](https://mlsanity.com/paper/2411.01220/) (2024)
- [Mechanistic Permutability: Match Features Across Layers](https://mlsanity.com/paper/2410.07656/) (2024)
- [Residual Stream Analysis with Multi-Layer SAEs](https://mlsanity.com/paper/2409.04185/) (2024)
- [Evaluating Sparse Autoencoders on Targeted Concept Erasure Tasks](https://mlsanity.com/paper/2411.18895/) (2024)
- [Formation of Representations in Neural Networks](https://mlsanity.com/paper/2410.03006/) (2024)

## Built on by (most cited first, 12 of 12)

- [Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?](https://mlsanity.com/paper/2602.14111/) (2026)
- [The Dead Salmons of AI Interpretability](https://mlsanity.com/paper/2512.18792/) (2025)
- [The Anatomy of Alignment: Decomposing Preference Optimization by Steering Sparse Features](https://mlsanity.com/paper/2509.12934/) (2025)
- [Toward Identifiable Sparse Autoencoders](https://mlsanity.com/paper/2605.31245/) (2026)
- [Sparse Autoencoders Encode Both Concepts and Functions: The Downstream Geometry of Feature Effects](https://mlsanity.com/paper/2607.24645/) (2026)
- [From Concept-Aligned Tokens to Vulnerable Features: Mechanistic Localization of Jailbreaks](https://mlsanity.com/paper/2604.23130/) (2026)
- [Sign-Aware Gated Sparse Autoencoders: Modeling Anticorrelated Features with Bi-Jump-ReLU Activations](https://mlsanity.com/paper/2605.28149/) (2026)
- [Building Fast, Evaluating Slow: Pipeline Choices Dominate Autointerpretability Score Variance](https://mlsanity.com/paper/2607.19386/) (2026)
- [The Misery of Mechanistic Interpretability: A Formal Perspective](https://mlsanity.com/paper/2609.15533/) (2026)
- [Representational Simplicity and Circuit Size Dissociate in a Threshold-Dependent Way: A Controlled Test via Adversarial Training](https://mlsanity.com/paper/2609.35890/) (2026)
- [Interpretable but Fragile? Robustness of Concept Bottlenecks under Geometric-Semantic Perturbations](https://mlsanity.com/paper/2609.38625/) (2026)
- [Backdooring Sparse Autoencoders](https://mlsanity.com/paper/2610.06049/) (2026)

## Abstract (arXiv)

Sparse autoencoders (SAEs) are commonly used to interpret the internal activations of large language models (LLMs) by mapping them to human-interpretable concept representations. While existing evaluations of SAEs focus on metrics such as the reconstruction-sparsity tradeoff, human (auto-)interpretability, and feature disentanglement, they overlook a critical aspect: the robustness of concept representations to input perturbations. We argue that robustness must be a fundamental consideration for concept representations, reflecting the fidelity of concept labeling. To this end, we formulate robustness quantification as input-space optimization problems and develop a comprehensive evaluation framework featuring realistic scenarios in which adversarial perturbations are crafted to manipulate SAE representations. Empirically, we find that tiny adversarial input perturbations can effectively manipulate concept-based interpretations in most scenarios without notably affecting the base LLM's activations. Overall, our results suggest that SAE concept representations are fragile and without further denoising or postprocessing they might be ill-suited for applications in model monitoring and oversight.