# Benchmarking Robustness of Multimodal Image-Text Models under Distribution Shift

arXiv: https://arxiv.org/abs/2212.08044 · HTML: https://mlsanity.com/paper/2212.08044/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Dec 2022.

## Built on by (most cited first, 24 of 24)

- [R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?](https://mlsanity.com/paper/2410.05474/) (2024)
- [Benchmarking Large Multimodal Models against Common Corruptions](https://mlsanity.com/paper/2401.11943/) (2024)
- [Image retrieval outperforms diffusion models on data augmentation](https://mlsanity.com/paper/2304.10253/) (2023)
- [Evaluating the Fairness of Discriminative Foundation Models in Computer Vision](https://mlsanity.com/paper/2310.11867/) (2023)
- [Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models](https://mlsanity.com/paper/2308.03151/) (2023)
- [Test-time Adaptation for Cross-modal Retrieval with Query Shift](https://mlsanity.com/paper/2410.15624/) (2024)
- [Visual Data-Type Understanding does not emerge from Scaling Vision-Language Models](https://mlsanity.com/paper/2310.08577/) (2023)
- [Rainbow Noise: Stress-Testing Multimodal Harmful-Meme Detectors on LGBTQ Content](https://mlsanity.com/paper/2507.19551/) (2025)
- [Visual Instruction Bottleneck Tuning](https://mlsanity.com/paper/2505.13946/) (2025)
- [ContextRef: Evaluating Referenceless Metrics For Image Description Generation](https://mlsanity.com/paper/2309.11710/) (2023)
- [Evaluating Durability: Benchmark Insights into Multimodal Watermarking](https://mlsanity.com/paper/2406.03728/) (2024)
- [Mapping the Mind of an Instruction-based Image Editing using SMILE](https://mlsanity.com/paper/2412.16277/) (2024)
- [Automatic benchmarking of large multimodal models via iterative experiment programming](https://mlsanity.com/paper/2406.12321/) (2024)
- [How does Watermarking Affect Visual Language Models in Document Understanding?](https://mlsanity.com/paper/2504.01048/) (2025)
- [Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency](https://mlsanity.com/paper/2505.14405/) (2025)
- [Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?](https://mlsanity.com/paper/2606.08894/) (2026)
- [VisMoDAl: Visual Analytics for Evaluating and Improving Corruption Robustness of Vision-Language Models](https://mlsanity.com/paper/2509.14571/) (2025)
- [PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applications](https://mlsanity.com/paper/2509.23879/) (2025)
- [Language-Guided Invariance Probing of Vision-Language Models](https://mlsanity.com/paper/2511.13494/) (2025)
- [Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning](https://mlsanity.com/paper/2608.29278/) (2026)
- [Efficient 3D-Aware Facial Image Editing via Attribute-Specific Prompt Learning](https://mlsanity.com/paper/2406.04413/) (2024)
- [Semantic Misalignment in Vision-Language Models under Perceptual Degradation](https://mlsanity.com/paper/2601.08355/) (2026)
- [An Attention Mechanism for Robust Multimodal Integration in a Global Workspace Architecture](https://mlsanity.com/paper/2602.08597/) (2026)
- [Hystar: Hypernetwork-driven Style-adaptive Retrieval via Dynamic SVD Modulation](https://mlsanity.com/paper/2605.10009/) (2026)

## Abstract (arXiv)

Multimodal image-text models have shown remarkable performance in the past few years. However, evaluating robustness against distribution shifts is crucial before adopting them in real-world applications. In this work, we investigate the robustness of 12 popular open-sourced image-text models under common perturbations on five tasks (image-text retrieval, visual reasoning, visual entailment, image captioning, and text-to-image generation). In particular, we propose several new multimodal robustness benchmarks by applying 17 image perturbation and 16 text perturbation techniques on top of existing datasets. We observe that multimodal models are not robust to image and text perturbations, especially to image perturbations. Among the tested perturbation methods, character-level perturbations constitute the most severe distribution shift for text, and zoom blur is the most severe shift for image data. We also introduce two new robustness metrics (MMI for MultiModal Impact score and MOR for Missing Object Rate) for proper evaluations of multimodal models. We hope our extensive study sheds light on new directions for the development of robust multimodal models. More details can be found on the project webpage: urlhttps://MMRobustness.github.io.