MIMIC: Multimodal Inversion for Model Interpretation and Conceptualization
On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning
Abstract (arXiv)
Vision Language Models (VLMs) encode multimodal inputs over large, complex, and difficult-to-interpret architectures, which limit transparency and trust. We propose a Multimodal Inversion for Model Interpretation and Conceptualization (MIMIC) framework that inverts the internal encodings of VLMs. MIMIC uses a joint VLM-based inversion and a feature alignment objective to account for VLM's autoregressive processing. It additionally includes a triplet of regularizers for spatial alignment, natural image smoothness, and semantic realism. We evaluate MIMIC both quantitatively and qualitatively by inverting visual concepts across a range of free-form VLM outputs of varying length. Reported results include both standard visual quality metrics and semantic text-based metrics. To the best of our knowledge, this is the first model inversion approach addressing visual interpretations of VLM concepts.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 54 references that are in the map. First posted on arXiv 11 Aug 2025.
Builds on
45 of 54 references.
Closest ancestors · 3
Same direction, newest first
- 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
- 2024Probing Multimodal Large Language Models for Global and Local Semantic Representations27 Feb
- 2023Improved Baselines with Visual Instruction Tuning5 Oct
Other references · 42
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
30 more
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2017Axiomatic Attribution for Deep Networkscited by 3,384
- 2023Mistral 7Bcited by 3,164
- 2016Improved Techniques for Training GANscited by 2,970
- 2022Locating and Editing Factual Associations in GPTcited by 2,691
- 2015Learning Deep Features for Discriminative Localizationcited by 2,436
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
- 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
- 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,416
- 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 675
- 2019Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networkscited by 582
- 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
- 2015Understanding Neural Networks Through Deep Visualizationcited by 334
- 2023Studying Large Language Model Generalization with Influence Functionscited by 326
- 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
- 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 178
- 2019Effectively Unbiased FID and Inception Score and where to find themcited by 165
- 2024SelfIE: Self-Interpretation of Large Language Model Embeddingscited by 67
- 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
- 2022GradViT: Gradient Inversion of Vision Transformerscited by 53
- 2024Interpreting the Second-Order Effects of Neurons in CLIPcited by 36
- 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 30
- 2025From Colors to Classes: Emergence of Concepts in Vision Transformerscited by 23
- 2024FaithLM: Towards Faithful Explanations for Large Language Modelscited by 19
- 2024Linear Explanations for Individual Neuronscited by 19
- 2022Plug-In Inversion: Model-Agnostic Inversion for Vision with Data Augmentationscited by 18
- 2020MANGO: A Python Library for Parallel Hyperparameter Tuningcited by 8
- 2025Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIPcited by 8
- 2021The Mind's Eye: Visualizing Class-Agnostic Features of CNNscited by 2
- 2023Leaping Into Memories: Space-Time Deep Feature Synthesiscited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modes
- 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability
- 2026VLMs Need Words: Vision Language Models Ignore Visual Detail In Favor of Semantic Anchors
- 2026VoT: Vision-of-Thought for Unified Multimodal Representation Alignment
- 2025Visual Representation Alignment for Multimodal Large Language Models
- 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuning
- 2026Vision Inference Former: Sustaining Visual Consistency in Multimodal Large Language Models
- 2026Measuring Cross-Modal Synergy: A Benchmark for VLM Explainability