mlsanityData to 14 Oct 2026

Decompose the model: Mechanistic interpretability in image models with Generalized Integrated Gradients (GIG)

arXiv 2409.01610 ↗First posted 3 Sep 2024cs.CV cs.AIcited by 1

On the map: Mechanistic interpretability → Interpreting vision and multimodal models

Abstract (arXiv)

In the field of eXplainable AI (XAI) in language models, the progression from local explanations of individual decisions to global explanations with high-level concepts has laid the groundwork for mechanistic interpretability, which aims to decode the exact operations. However, this paradigm has not been adequately explored in image models, where existing methods have primarily focused on class-specific interpretations. This paper introduces a novel approach to systematically trace the entire pathway from input through all intermediate layers to the final output within the whole dataset. We utilize Pointwise Feature Vectors (PFVs) and Effective Receptive Fields (ERFs) to decompose model embeddings into interpretable Concept Vectors. Then, we calculate the relevance between concept vectors with our Generalized Integrated Gradients (GIG), enabling a comprehensive, dataset-wide analysis of model behavior. We validate our method of concept extraction and concept attribution in both qualitative and quantitative evaluations. Our approach advances the understanding of semantic significance within image models, offering a holistic view of their operational mechanics.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Interpreting vision and multimodal models, one of 13 directions in Mechanistic interpretability. Builds on 16 of its 20 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

16 of 20 references.

References · 16

  1. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  2. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  6. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  7. 2015Explaining NonLinear Classification Decisions with Deep Taylor Decompositioncited by 250
  8. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 178
  9. 2022From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagationcited by 141
  10. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 111
  11. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 109
  12. 2024Universal Neurons in GPT2 Language Modelscited by 102
4 more
  1. 2022Global Concept-Based Interpretability for Graph Neural Networks via Neuron Analysiscited by 32
  2. 2022Causal Proxy Models for Concept-Based Model Explanationscited by 29
  3. 2024Visual Concept Connectome (VCC): Open World Concept Discovery and their Interlayer Connections in Deep Modelscited by 22
  4. 2024Respect the model: Fine-grained and Robust Explanation with Sharing Ratio Decompositioncited by 6

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Bi-ICE: An Inner Interpretable Framework for Image Classification via Bi-directional Interactions between Concept and Input Embeddingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Scale Alone Does not Improve Mechanistic Interpretability in Vision Models
  2. 2026From Local to Global to Mechanistic: An iERF-Centered Unified Framework for Interpreting Vision Models
  3. 2024LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models
  4. 2024VALE: A Multimodal Visual and Language Explanation Framework for Image Classifiers using eXplainable AI and Language Models
  5. 2026Clarity: The Flexibility-Interpretability Trade-Off in Sparsity-aware Concept Bottleneck Models
  6. 2025Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition
  7. 2026DEX-AR: A Dynamic Explainability Method for Autoregressive Vision-Language Models
  8. 2025CE-FAM: Concept-Based Explanation via Fusion of Activation Maps