# ICYM2I: The illusion of multimodal informativeness under missingness

arXiv: https://arxiv.org/abs/2505.16953 · HTML: https://mlsanity.com/paper/2505.16953/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 107 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [Tensor Fusion Network for Multimodal Sentiment Analysis](https://mlsanity.com/paper/1707.07250/) (2017)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [Multimodal Deep Learning](https://mlsanity.com/paper/2301.04856/) (2023)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [MultiBench: Multiscale Benchmarks for Multimodal Representation Learning](https://mlsanity.com/paper/2107.07502/) (2021)
- [UR-FUNNY: A Multimodal Language Dataset for Understanding Humor](https://mlsanity.com/paper/1904.06618/) (2019)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Why are Visually-Grounded Language Models Bad at Image Classification?](https://mlsanity.com/paper/2405.18415/) (2024)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [ECG-FM: An Open Electrocardiogram Foundation Model](https://mlsanity.com/paper/2408.05178/) (2024)
- [GCNet: Graph Completion Network for Incomplete Multimodal Learning in Conversation](https://mlsanity.com/paper/2203.02177/) (2022)
- [Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework](https://mlsanity.com/paper/2302.12247/) (2023)
- [Multimodal Object Detection via Probabilistic Ensembling](https://mlsanity.com/paper/2104.02904/) (2021)
- [POG: Personalized Outfit Generation for Fashion Recommendation at Alibaba iFashion](https://mlsanity.com/paper/1905.01866/) (2019)
- [Graphical Models for Processing Missing Data](https://mlsanity.com/paper/1801.03583/) (2018)
- [M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities](https://mlsanity.com/paper/2303.05302/) (2023)
- [Towards Good Practices for Missing Modality Robust Action Recognition](https://mlsanity.com/paper/2211.13916/) (2022)
- [ELIXR: Towards a general purpose X-ray artificial intelligence system through alignment of large language models and radiology vision encoders](https://mlsanity.com/paper/2308.01317/) (2023)
- [Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2404.16456/) (2024)

## Built on by (most cited first, 1 of 1)

- [A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models](https://mlsanity.com/paper/2603.29676/) (2026)

## Abstract (arXiv)

Multimodal learning is of continued interest in artificial intelligence-based applications, motivated by the potential information gain from combining different data modalities. However, modalities observed in the source environment may differ from the modalities observed in the target environment due to multiple factors, including cost, hardware failure, or the perceived textitinformativeness of a given modality. This change in missingness patterns between the source and target environment has not been carefully studied. Na"ive estimation of the information gain associated with including an additional modality without accounting for missingness may result in improper estimates of that modality's value in the target environment. We formalize the problem of missingness, demonstrate its ubiquity, and show that the subsequent distribution shift induces bias when the missingness process is not explicitly accounted for. To address this issue, we introduce ICYM2I (In Case You Multimodal Missed It), a framework for the evaluation of predictive performance and information gain under missingness through inverse probability weighting-based correction. We demonstrate the importance of the proposed adjustment to estimate information gain under missingness on synthetic, semi-synthetic, and real-world datasets.