mlsanityData to 14 Oct 2026

A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models

arXiv 2502.17516 ↗First posted 22 Feb 2025cs.LG cs.AIcited by 34

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

The rise of foundation models has transformed machine learning research, prompting efforts to uncover their inner workings and develop more efficient and reliable applications for better control. While significant progress has been made in interpreting Large Language Models (LLMs), multimodal foundation models (MMFMs) - such as contrastive vision-language models, generative vision-language models, and text-to-image models - pose unique interpretability challenges beyond unimodal frameworks. Despite initial studies, a substantial gap remains between the interpretability of LLMs and MMFMs. This survey explores two key aspects: (1) the adaptation of LLM interpretability methods to multimodal models and (2) understanding the mechanistic differences between unimodal language models and crossmodal systems. By systematically reviewing current MMFM analysis techniques, we propose a structured taxonomy of interpretability methods, compare insights across unimodal and multimodal architectures, and highlight critical research gaps.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 136 of its 180 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

136 of 180 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey3 Dec
  2. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning21 Jun
  3. 2024A Concept-Based Explainability Framework for Large Multimodal Models12 Jun
  4. 2024Probing Multimodal Large Language Models for Global and Local Semantic Representations27 Feb
  5. 2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers13 Nov
  6. 2023Generating Images with Multimodal Language Models26 May

Other references · 130

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  8. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  9. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  11. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
38 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
  5. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  7. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  8. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  10. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  11. 2022Mass-Editing Memory in a Transformercited by 1,001
  12. 2019Attention is not Explanationcited by 929
  13. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  14. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  15. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  16. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  17. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  18. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  19. 2019Attention is not not Explanationcited by 636
  20. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  21. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  22. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  23. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  24. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  25. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  26. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  27. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  28. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  29. 2023In-Context Learning Creates Task Vectorscited by 312
  30. 2019Interpretable machine learning: definitions, methods, and applicationscited by 305
  31. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 302
  32. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  33. 2022Inversion-Based Style Transfer with Diffusion Modelscited by 291
  34. 2023TRAK: Attributing Model Behavior at Scalecited by 283
  35. 2013Direct and Indirect Effectscited by 280
  36. 2020Influence Functions in Deep Learning Are Fragilecited by 252
  37. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  38. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 236

50 most cited of 130 All 130 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 19.

34 papers.

Sort:

By area:

  1. 2026Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Modelscited by 27
  2. 2025LLM-Powered AI Agent Systems and Their Applications in Industrycited by 22
  3. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 20
  4. 2025BiasEdit: Debiasing Stereotyped Language Models via Model Editingcited by 18
  5. 2025Vision Function Layer in Multimodal LLMscited by 17
  6. 2025Where MLLMs Attend and What They Rely On: Explaining Autoregressive Token Generationcited by 16
  7. 2025Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrationscited by 15
  8. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 11
  9. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Modelscited by 8
  10. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradientscited by 8
  11. 2025GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMscited by 6
  12. 2025Evaluation of Geolocation Capabilities of Multimodal Large Language Models and Analysis of Associated Privacy Riskscited by 3
22 more
  1. 2025Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolismcited by 3
  2. 2026Does AI See like Art Historians? Interpreting How Vision Language Models Recognize Artistic Stylecited by 3
  3. 2025I Spy With My Model's Eye: Visual Search as a Behavioural Test for MLLMscited by 2
  4. 2026SPD-Faith Bench: Diagnosing and Improving Faithfulness in Chain-of-Thought for Multimodal Large Language Modelscited by 2
  5. 2026The physics of AI weather modelscited by 2
  6. 2026MATEX: Multi-scale Attention and Text-guided Explainability of Medical Vision-Language Modelscited by 1
  7. 2026Backdoor Directions in Vision Transformerscited by 1
  8. 2026From Heads to Neurons: Causal Attribution and Steering in Multi-Task Vision-Language Modelscited by 1
  9. 2026Do Factual Recall Mechanisms Carry over from Text to Speech in Multimodal Language Models?cited by 1
  10. 2026Which Modality Decides? Counterfactual Modality Attribution for Multimodal LLMscited by 1
  11. 2025Beyond the Linear Separability Ceiling: Aligning Representations in VLMscited by 0
  12. 2025Interpret, prune and distill Donut : towards lightweight VLMs for VQA on documentcited by 0
  13. 2025Understanding Temporal Logic Consistency in Video-Language Models through Cross-Modal Attention Discriminabilitycited by 0
  14. 2026Structural Graph Probing of Vision-Language Modelscited by 0
  15. 2026Breaking Bad: Interpretability-Based Safety Audits of State-of-the-Art LLMscited by 0
  16. 2026SimReg: Achieving Higher Performance in the Pretraining via Embedding Similarity Regularizationcited by 0
  17. 2026Unsupervised Features Mining via Activation Geometrycited by 0
  18. 2026Through the LENS: Local Geometric Decomposition of Vision-Language Model Representationscited by 0
  19. 2026Multimodal Model Diffing for Feature Discovery and Controlcited by 0
  20. 2026Beyond Tokens: A Survey on Decoding Methods for Large Language and Vision-Language Modelscited by 0
  21. 2026Vision-language models know more about agriculture than they show and rubric-grounded verifications close the gapcited by 0
  22. 2026Mechanism-Level Evaluation for Vision-Language Models: Controlled Activation-Replacement Diagnosis of Gender Biascited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistants
  2. 2025SAE-V: Interpreting Multimodal Models for Enhanced Alignment
  3. 2024The Visualization JUDGE : Can Multimodal Foundation Models Guide Visualization Design Through Visual Perception?
  4. 2024LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models
  5. 2026Reliable and Responsible Foundation Models: A Comprehensive Survey
  6. 2025Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data
  7. 2024Cross-Modal Projection in Multimodal LLMs Doesn't Really Project Visual Attributes to Textual Space
  8. 2025Beyond Cross-Modal Alignment: Measuring and Leveraging Modality Gap in Vision-Language Models