# Evaluating Robustness of Vision-Language Models Under Noisy Conditions

arXiv: https://arxiv.org/abs/2509.12492 · HTML: https://mlsanity.com/paper/2509.12492/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 18 of 18)

- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [nocaps: novel object captioning at scale](https://mlsanity.com/paper/1812.08658/) (2018)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks](https://mlsanity.com/paper/1711.07064/) (2017)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [ALIP: Adaptive Language-Image Pre-training with Synthetic Caption](https://mlsanity.com/paper/2308.08428/) (2023)
- [CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching](https://mlsanity.com/paper/2404.03653/) (2024)
- [Towards Diverse and Natural Image Descriptions via a Conditional GAN](https://mlsanity.com/paper/1703.06029/) (2017)
- [Transfer CLIP for Generalizable Image Denoising](https://mlsanity.com/paper/2403.15132/) (2024)
- [Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective](https://mlsanity.com/paper/2404.19287/) (2024)
- [RoRA-VLM: Robust Retrieval-Augmented Vision Language Models](https://mlsanity.com/paper/2410.08876/) (2024)
- [R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interest](https://mlsanity.com/paper/2410.20327/) (2024)
- [Visual Language Model based Cross-modal Semantic Communication Systems](https://mlsanity.com/paper/2407.00020/) (2024)
- [RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models](https://mlsanity.com/paper/2304.10727/) (2023)
- [Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training](https://mlsanity.com/paper/1703.10476/) (2017)

## Abstract (arXiv)

Vision-Language Models (VLMs) have attained exceptional success across multimodal tasks such as image captioning and visual question answering. However, their robustness under noisy conditions remains unfamiliar. In this study, we present a comprehensive evaluation framework to evaluate the performance of several state-of-the-art VLMs under controlled perturbations, including lighting variation, motion blur, and compression artifacts. We used both lexical-based metrics (BLEU, METEOR, ROUGE, CIDEr) and neural-based similarity measures using sentence embeddings to quantify semantic alignment. Our experiments span diverse datasets, revealing key insights: (1) descriptiveness of ground-truth captions significantly influences model performance; (2) larger models like LLaVA excel in semantic understanding but do not universally outperform smaller models; and (3) certain noise types, such as JPEG compression and motion blur, dramatically degrade performance across models. Our findings highlight the nuanced trade-offs between model size, dataset characteristics, and noise resilience, offering a standardized benchmark for future robust multimodal learning.