# SPARC: Concept-Aligned Sparse Autoencoders for Cross-Model and Cross-Modal Interpretability

arXiv: https://arxiv.org/abs/2507.06265 · HTML: https://mlsanity.com/paper/2507.06265/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Interpreting vision and multimodal models, one of 13 directions in Mechanistic interpretability.
- First posted on arXiv 7 Jul 2025.

## Placed in

- Mechanistic interpretability / [Interpreting vision and multimodal models](https://mlsanity.com/mechinterp/interpreting-vision-and-multimodal-models/)

## Built on by (most cited first, 9 of 9)

- [Interpretable and Steerable Concept Bottleneck Sparse Autoencoders](https://mlsanity.com/paper/2512.10805/) (2025)
- [Mechanistic Analysis of Alignment Algorithms in Language Models](https://mlsanity.com/paper/2606.09850/) (2026)
- [One Adapter Pair per Model: A Universal Activation Interface for Language Models](https://mlsanity.com/paper/2608.09521/) (2026)
- [MonoLoss: A Training Objective for Interpretable Monosemantic Representations](https://mlsanity.com/paper/2602.12403/) (2026)
- [When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities](https://mlsanity.com/paper/2607.08605/) (2026)
- [Mechanistically Interpreting Compression in Vision-Language Models](https://mlsanity.com/paper/2603.25035/) (2026)
- [SharedSAE: One Feature Dictionary Across Language Models](https://mlsanity.com/paper/2609.04344/) (2026)
- [Cross-Modal Latent Influence: Patch-Level Training Sample Attribution for Vision-Language Models](https://mlsanity.com/paper/2610.14190/) (2026)
- [MP-MNCA: Morphology-Prior Masked Neighbor Cross-Attention for Efficient Spatial Domain Identification](https://mlsanity.com/paper/2610.16212/) (2026)

## Abstract (arXiv)

Understanding how different AI models encode the same high-level concepts, such as objects or attributes, remains challenging because each model typically produces its own isolated representation. Existing interpretability methods like Sparse Autoencoders (SAEs) produce latent concepts individually for each model, resulting in incompatible concept spaces and limiting cross-model interpretability. To address this, we introduce SPARC (Sparse Autoencoders for Aligned Representation of Concepts), a new framework that learns a single, unified latent space shared across diverse architectures and modalities (e.g., vision models like DINO, and multimodal models like CLIP). SPARC's alignment is enforced through two key innovations: (1) a Global TopK sparsity mechanism, ensuring all input streams activate identical latent dimensions for a given concept; and (2) a Cross-Reconstruction Loss, which explicitly encourages semantic consistency between models. On Open Images, SPARC dramatically improves concept alignment, achieving a Jaccard similarity of 0.80, more than tripling the alignment compared to previous methods. SPARC creates a shared sparse latent space where individual dimensions often correspond to similar high-level concepts across models and modalities, enabling direct comparison of how different architectures represent identical concepts without requiring manual alignment or model-specific analysis. As a consequence of this aligned representation, SPARC also enables practical applications such as text-guided spatial localization in vision-only models and cross-model/cross-modal retrieval. Code and models are available at https://github.com/AtlasAnalyticsLab/SPARC