mlsanityData to 14 Oct 2026

Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment

arXiv 2502.03714 ↗First posted 6 Feb 2025cs.CV cs.LGcited by 62

On the map: Mechanistic interpretability → Interpreting vision and multimodal models

Abstract (arXiv)

We present Universal Sparse Autoencoders (USAEs), a framework for uncovering and aligning interpretable concepts spanning multiple pretrained deep neural networks. Unlike existing concept-based interpretability methods, which focus on a single model, USAEs jointly learn a universal concept space that can reconstruct and interpret the internal activations of multiple models at once. Our core insight is to train a single, overcomplete sparse autoencoder (SAE) that ingests activations from any model and decodes them to approximate the activations of any other model under consideration. By optimizing a shared objective, the learned dictionary captures common factors of variation-concepts-across different tasks, architectures, and datasets. We show that USAEs discover semantically coherent and important universal concepts across vision models; ranging from low-level features (e.g., colors and textures) to higher-level structures (e.g., parts and objects). Overall, USAEs provide a powerful new method for interpretable cross-model analysis and offers novel applications, such as coordinated activation maximization, that open avenues for deeper insights in multi-model AI systems

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Placed in Interpreting vision and multimodal models, one of 13 directions in Mechanistic interpretability. Builds on 66 of its 98 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

66 of 98 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Understanding Video Transformers via Universal Concept Discovery19 Jan

Other references · 65

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  7. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  8. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  9. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  10. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  11. 2013Describing Textures in the Wildcited by 2,125
  12. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
38 more
  1. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  2. 2017SmoothGrad: removing noise by adding noisecited by 1,109
  3. 2020Concept Bottleneck Modelscited by 1,032
  4. 2014Striving for Simplicity: The All Convolutional Netcited by 990
  5. 2022Toy Models of Superpositioncited by 971
  6. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 903
  7. 2018Robustness May Be at Odds with Accuracycited by 880
  8. 2023Vision Transformers Need Registerscited by 871
  9. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  10. 2024Scaling and evaluating sparse autoencoderscited by 553
  11. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  12. 2024The Platonic Representation Hypothesiscited by 382
  13. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  14. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  15. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 259
  16. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  17. 2022Relative representations enable zero-shot latent space communicationcited by 191
  18. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 190
  19. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 178
  20. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoderscited by 160
  21. 2023Getting aligned on representational alignmentcited by 143
  22. 2020Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectorscited by 120
  23. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 119
  24. 2023Foundation Models and Fair Usecited by 116
  25. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 109
  26. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  27. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  28. 2019When Explanations Lie: Why Many Modified BP Attributions Failcited by 92
  29. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  30. 2021The effectiveness of feature attribution methods and its correlation with automatic evaluation scorescited by 88
  31. 2023White-Box Transformers via Sparse Rate Reductioncited by 80
  32. 2022What do Vision Transformers Learn? A Visual Explorationcited by 76
  33. 2019Understanding Neural Networks via Feature Visualization: A surveycited by 65
  34. 2021Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysiscited by 65
  35. 2020Evaluations and Methods for Explanation through Robustness Analysiscited by 51
  36. 2023Multi-dimensional concept discovery (MCD): A unifying framework with completeness guaranteescited by 50
  37. 2014Sparse Modeling for Image and Vision Processingcited by 48
  38. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 48

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 24; 2026: 38.

62 papers.

Sort:

By area:

  1. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 73
  2. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 71
  3. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 61
  4. 2025Overcoming Sparsity Artifacts in Crosscoders to Interpret Chat-Tuningcited by 34
  5. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  6. 2025Interpretable and Testable Vision Features via Sparse Autoencoderscited by 25
  7. 2025From superposition to sparse codes: interpretable representations in neural networkscited by 16
  8. 2025On the Limits of Sparse Autoencoders: A Theoretical Framework and Reweighted Remedycited by 14
  9. 2025AbsTopK: Rethinking Sparse Autoencoders For Bidirectional Featurescited by 9
  10. 2025From What to How: Attributing CLIP's Latent Components Reveals Unexpected Semantic Reliancecited by 6
  11. 2025SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutionscited by 6
  12. 2025Interpretable and Steerable Concept Bottleneck Sparse Autoencoderscited by 6
38 more
  1. 2026Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifoldscited by 5
  2. 2026Learning Concept Bottleneck Models from Mechanistic Explanationscited by 4
  3. 2025Mind the (Data) Gap: Evaluating Vision Systems in Small Data Applicationscited by 3
  4. 2025Interpreting CFD Surrogates through Sparse Autoencoderscited by 3
  5. 2025Atlas-Alignment: Making Interpretability Transferable Across Language Modelscited by 3
  6. 2026Towards Interpretable Hallucination Analysis and Mitigation in LVLMs via Contrastive Neuron Steeringcited by 3
  7. 2026Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Modelscited by 3
  8. 2026Inside the Visual Mind: Neuroscience-Motivated Concept Circuits for Interpreting and Steering Vision Transformerscited by 3
  9. 2025Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steeringcited by 2
  10. 2025Cross-Model Semantics in Representation Learningcited by 2
  11. 2025Position: Require Frontier AI Labs To Release Small "Analog" Modelscited by 2
  12. 2026Decomposing multimodal embedding spaces with group-sparse autoencoderscited by 2
  13. 2026Characterizing Universal Object Representations Across Vision Modelscited by 2
  14. 2026One Adapter Pair per Model: A Universal Activation Interface for Language Modelscited by 2
  15. 2025Explaining How Visual, Textual and Multimodal Encoders Share Conceptscited by 1
  16. 2025MIMIC: Multimodal Inversion for Model Interpretation and Conceptualizationcited by 1
  17. 2025Zero-Shot Textual Explanations via Translating Decision-Critical Featurescited by 1
  18. 2026LUCID-SAE: Learning Unified Vision-Language Sparse Codes for Interpretable Concept Discoverycited by 1
  19. 2026Sparse Autoencoders as a Steering Basis for Phase Synchronization in Graph-Based CFD Surrogatescited by 1
  20. 2026Beyond Semantics: Disentangling Information Scope in Sparse Autoencoders for CLIPcited by 1
  21. 2026A Distributional View for Visual Mechanistic Interpretability: KL-Minimal Soft-Constraint Principlecited by 1
  22. 2026Where Concept Erasure Should Occur: Concept-Layer Alignment in Text-to-Video Diffusion Modelscited by 1
  23. 2026A Unifying Framework for Concept-Based Representational Similaritycited by 1
  24. 2026When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalitiescited by 1
  25. 2025Representational Difference Explanationscited by 0
  26. 2025Features Emerge as Discrete States: The First Application of SAEs to 3D Representationscited by 0
  27. 2025The Deleuzian Representation Hypothesiscited by 0
  28. 2026Can Cross-Layer Transcoders Replace Vision Transformer Activations? An Interpretable Perspective on Visioncited by 0
  29. 2026Capability neq Interpretability: Human Interpretability of Vision Foundation Modelscited by 0
  30. 2026Steered Generation via Gradient-Based Optimization on Sparse Query Featurescited by 0
  31. 2026Similarity-based representation factorization for revealing interpretable dimensions in representational datacited by 0
  32. 2026Robust and Generalizable Safety Steering for Text-to-Image Diffusion Transformerscited by 0
  33. 2026The Cross-Architecture Substrate: A Domain-Transcendent, Calibration-Surviving Geometric Invariant of Modern Vision Encoderscited by 0
  34. 2026Contrastive-Difference CKA Reveals Concept-Specific Structural Alignment Across Language Model Architecturescited by 0
  35. 2026Cross-seed explainability using Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoderscited by 0
  36. 2026Sparse Autoencoders for Interpretable Out-of-Distribution Detectioncited by 0
  37. 2026Through the LENS: Local Geometric Decomposition of Vision-Language Model Representationscited by 0
  38. 2026SAE-Xplainers: Rule-Based Feature Interpretation for Extreme Earth Eventscited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SPARC: Concept-Aligned Sparse Autoencoders for Cross-Model and Cross-Modal Interpretability
  2. 2026Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations
  3. 2025Probing the Representational Power of Sparse Autoencoders in Vision Models
  4. 2026Supervised sparse auto-encoders for interpretable and compositional representations
  5. 2026Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs
  6. 2026Steering Vision-Language Models with Joint Sparse Autoencoders
  7. 2026Do Sparse Autoencoders Learn Meaningful Concept Hierarchies?
  8. 2026Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders