mlsanityData to 7 Oct 2026

A Concept-Based Explainability Framework for Large Multimodal Models

arXiv 2406.08074 ↗First posted 12 Jun 2024cs.LG cs.AI cs.CLcited by 39

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMMs) combine unimodal encoders and large language models (LLMs) to perform multimodal tasks. Despite recent advancements towards the interpretability of these models, understanding internal representations of LMMs remains largely a mystery. In this paper, we present a novel framework for the interpretation of LMMs. We propose a dictionary learning based approach, applied to the representation of tokens. The elements of the learned dictionary correspond to our proposed concepts. We show that these concepts are well semantically grounded in both vision and text. Thus we refer to these as ``multi-modal concepts''. We qualitatively and quantitatively evaluate the results of the learnt concepts. We show that the extracted multimodal concepts are useful to interpret representations of test samples. Finally, we evaluate the disentanglement between different concepts and the quality of grounding concepts visually and textually. Our code is publicly available at https://github.com/mshukor/xl-vlms

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 50 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

40 of 50 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers13 Nov
  2. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  10. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  12. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
26 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  5. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  6. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  7. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  8. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  9. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 177
  10. 2022Linearly Mapping from Image to Text Spacecited by 139
  11. 2020Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectorscited by 119
  12. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  13. 2023A Holistic Approach to Unifying Automatic Concept Extraction and Concept Importance Estimationcited by 108
  14. 2021What I Cannot Predict, I Do Not Understand: A Human-Centered Evaluation Framework for Explainability Methodscited by 98
  15. 2020The role of explainability in creating trustworthy artificial intelligence for health care: a comprehensive survey of the terminology, design choices, and evaluation strategiescited by 91
  16. 2021MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuningcited by 85
  17. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  18. 2023Multimodal Neurons in Pretrained Text-Only Transformerscited by 49
  19. 2023Multi-dimensional concept discovery (MCD): A unifying framework with completeness guaranteescited by 48
  20. 2022MAPL: Parameter-Efficient Adaptation of Unimodal Pre-Trained Models for Vision-Language Few-Shot Promptingcited by 41
  21. 2023eP-ALM: Efficient Perceptual Augmentation of Language Modelscited by 33
  22. 2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformerscited by 22
  23. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputscited by 22
  24. 2023Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanismcited by 21
  25. 2024Improved Baselines for Data-efficient Perceptual Augmentation of LLMscited by 18
  26. 2020Flexible and Context-Specific AI Explainability: A Multidisciplinary Approachcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 21; 2026: 11.

39 papers.

Sort:

By area:

  1. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  2. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Modelscited by 70
  3. 2025Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Modelscited by 67
  4. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
  5. 2024Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Surveycited by 52
  6. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  7. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  8. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  9. 2025Interpreting the linear structure of vision-language model embedding spacescited by 32
  10. 2024Large Multi-modal Models Can Interpret Features in Large Multi-modal Modelscited by 23
  11. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  12. 2024Concept-Based Explanations in Computer Vision: Where Are We and Where Could We Go?cited by 17
27 more
  1. 2025Head Pursuit: Probing Attention Specialization in Multimodal Transformerscited by 16
  2. 2025Analyzing Finetuning Representation Shift for Multimodal LLMs Steeringcited by 15
  3. 2025Learning to Steer: Input-dependent Steering for Multimodal LLMscited by 13
  4. 2025VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Setcited by 12
  5. 2025Re-Imagining Multimodal Instruction Tuning: A Representation Viewcited by 9
  6. 2025Probing the Representational Power of Sparse Autoencoders in Vision Modelscited by 9
  7. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  8. 2024MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Modelscited by 7
  9. 2026Bi-Orthogonal Factor Decomposition for Vision Transformerscited by 4
  10. 2025A Geometric Unification of Concept Learning with Concept Conescited by 3
  11. 2026Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamicscited by 3
  12. 2026Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifoldscited by 3
  13. 2024Open Llama2 Model for the Lithuanian Languagecited by 2
  14. 2025Causality Is Key to Understand and Balance Multiple Goals in Trustworthy ML and Foundation Modelscited by 2
  15. 2025Insights into a radiology-specialised multimodal large language model with sparse autoencoderscited by 2
  16. 2024TACO: Training-free Sound Prompted Segmentation via Semantically Constrained Audio-visual CO-factorizationcited by 1
  17. 2026One-shot Optimized Steering Vector for Hallucination Mitigation for VLMscited by 1
  18. 2026From Features to Actions: Explainability in Traditional and Agentic AI Systemscited by 1
  19. 2025Conceptualizing Uncertainty: A Concept-based Approach to Explaining Uncertaintycited by 0
  20. 2025Rationale-Enhanced Decoding for Multi-modal Chain-of-Thoughtcited by 0
  21. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  22. 2026Mind the Gap No More: Achieving Zero-Gap Multimodal Integration via One Tokenizercited by 0
  23. 2026Transferable XAI: Relating Understanding Across Domains with Explanation Transfercited by 0
  24. 2026Concept-Enhanced Multimodal RAG: Towards Interpretable and Accurate Radiology Report Generationcited by 0
  25. 2026Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Mediacited by 0
  26. 2026The Hyperspherical Geometry of CLIP Latent Space: A Semantic Mixture Modelcited by 0
  27. 2026From Plots to Words: Model-Aware Multimodal Explanations as a Foundation for Accessible, Non-Visual Interactioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unified Multimodal Understanding via Byte-Pair Visual Encoding
  2. 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
  3. 2025On Large Multimodal Models as Open-World Image Classifiers
  4. 2025Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data
  5. 2024Enhancing Multimodal Large Language Models Complex Reason via Similarity Computation
  6. 2025Large Language Models Facilitate Vision Reflection in Image Classification
  7. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  8. 2024Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion