# Explaining latent representations of generative models with large multimodal models

arXiv: https://arxiv.org/abs/2402.01858 · HTML: https://mlsanity.com/paper/2402.01858/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 26 references that are in the map.
- First posted on arXiv 2 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 17 of 17)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Score-based Generative Modeling in Latent Space](https://mlsanity.com/paper/2106.05931/) (2021)
- [Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2311.13165/) (2023)
- [Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning](https://mlsanity.com/paper/2401.06805/) (2024)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Models](https://mlsanity.com/paper/2401.00625/) (2024)
- [Diffusion-Based Representation Learning](https://mlsanity.com/paper/2105.14257/) (2021)
- [Learning Data Representations with Joint Diffusion Models](https://mlsanity.com/paper/2301.13622/) (2023)
- [Large Multimodal Models: Notes on CVPR 2023 Tutorial](https://mlsanity.com/paper/2306.14895/) (2023)
- [DOT-VAE: Disentangling One Factor at a Time](https://mlsanity.com/paper/2210.10920/) (2022)

## Built on by (most cited first, 3 of 3)

- [Decompose, Look, and Reason: Reinforced Latent Reasoning for VLMs](https://mlsanity.com/paper/2604.07518/) (2026)
- [LatentExplainer: Explaining Latent Representations in Deep Generative Models with Multimodal Large Language Models](https://mlsanity.com/paper/2406.14862/) (2024)
- [Towards Evaluating Robustness of Prompt Adherence in Text to Image Models](https://mlsanity.com/paper/2507.08039/) (2025)

## Abstract (arXiv)

Learning interpretable representations of data generative latent factors is an important topic for the development of artificial intelligence. With the rise of the large multimodal model, it can align images with text to generate answers. In this work, we propose a framework to comprehensively explain each latent variable in the generative models using a large multimodal model. We further measure the uncertainty of our generated explanations, quantitatively evaluate the performance of explanation generation among multiple large multimodal models, and qualitatively visualize the variations of each latent variable to learn the disentanglement effects of different generative models on explanations. Finally, we discuss the explanatory capabilities and limitations of state-of-the-art large multimodal models.