# Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling

arXiv: https://arxiv.org/abs/2602.16979 · HTML: https://mlsanity.com/paper/2602.16979/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 42 of its 55 references that are in the map.
- First posted on arXiv 19 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 42)

- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs](https://mlsanity.com/paper/2401.06209/) (2024)
- [BLINK: Multimodal Large Language Models Can See but Not Perceive](https://mlsanity.com/paper/2404.12390/) (2024)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answering](https://mlsanity.com/paper/1712.00377/) (2017)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Molecular geometry prediction using a deep generative graph neural network](https://mlsanity.com/paper/1904.00314/) (2019)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [Generalized Multimodal ELBO](https://mlsanity.com/paper/2105.02470/) (2021)
- [Beyond Question-Based Biases: Assessing Multimodal Shortcut Learning in Visual Question Answering](https://mlsanity.com/paper/2104.03149/) (2021)
- [Integrated multimodal artificial intelligence framework for healthcare applications](https://mlsanity.com/paper/2202.12998/) (2022)
- [What's a good imputation to predict with missing values?](https://mlsanity.com/paper/2106.00311/) (2021)
- [M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2303.05302/) (2023)
- [Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence](https://mlsanity.com/paper/2006.08242/) (2020)
- [Generative Models of Visually Grounded Imagination](https://mlsanity.com/paper/1705.10762/) (2017)
- [On the Limitations of Multimodal VAEs](https://mlsanity.com/paper/2110.04121/) (2021)
- [Training Strategies to Handle Missing Modalities for Audio-Visual Expression Recognition](https://mlsanity.com/paper/2010.00734/) (2020)
- [Perceptual Score: What Data Modalities Does Your Model Perceive?](https://mlsanity.com/paper/2110.14375/) (2021)
- [Assessing Modality Bias in Video Question Answering Benchmarks with Multimodal Large Language Models](https://mlsanity.com/paper/2408.12763/) (2024)
- [Classification of datasets with imputed missing values: does imputation quality matter?](https://mlsanity.com/paper/2206.08478/) (2022)
- [A Batch Normalized Inference Network Keeps the KL Vanishing Away](https://mlsanity.com/paper/2004.12585/) (2020)
- [Check It Again: Progressive Visual Question Answering via Visual Entailment](https://mlsanity.com/paper/2106.04605/) (2021)

## Abstract (arXiv)

Despite the recent success of Multimodal Large Language Models (MLLMs), existing approaches predominantly assume the availability of multiple modalities during training and inference. In practice, multimodal data is often incomplete because modalities may be missing, collected asynchronously, or available only for a subset of examples. In this work, we propose PRIMO, a supervised latent-variable imputation model that quantifies the predictive impact of any missing modality within the multimodal learning setting. PRIMO enables the use of all available training examples, whether modalities are complete or partial. Specifically, it models the missing modality through a latent variable that captures its relationship with the observed modality in the context of prediction. During inference, we draw many samples from the learned distribution over the missing modality to both obtain the marginal predictive distribution (for the purpose of prediction) and analyze the impact of the missing modalities on the prediction for each instance. We evaluate PRIMO on a synthetic XOR dataset, Audio-Vision MNIST, and MIMIC-III for mortality and ICD-9 prediction. Across all datasets, PRIMO obtains performance comparable to unimodal baselines when a modality is fully missing and to multimodal baselines when all modalities are available. PRIMO quantifies the predictive impact of a modality at the instance level using a variance-based metric computed from predictions across latent completions. We visually demonstrate how varying completions of the missing modality result in a set of plausible labels.