Evaluating Robustness of Vision-Language Models Under Noisy Conditions
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Vision-Language Models (VLMs) have attained exceptional success across multimodal tasks such as image captioning and visual question answering. However, their robustness under noisy conditions remains unfamiliar. In this study, we present a comprehensive evaluation framework to evaluate the performance of several state-of-the-art VLMs under controlled perturbations, including lighting variation, motion blur, and compression artifacts. We used both lexical-based metrics (BLEU, METEOR, ROUGE, CIDEr) and neural-based similarity measures using sentence embeddings to quantify semantic alignment. Our experiments span diverse datasets, revealing key insights: (1) descriptiveness of ground-truth captions significantly influences model performance; (2) larger models like LLaVA excel in semantic understanding but do not universally outperform smaller models; and (3) certain noise types, such as JPEG compression and motion blur, dramatically degrade performance across models. Our findings highlight the nuanced trade-offs between model size, dataset characteristics, and noise resilience, offering a standardized benchmark for future robust multimodal learning.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 31 references that are in the map. First posted on arXiv 15 Sep 2025.
Builds on
18 of 31 references.
Closest ancestors · 1
Same direction, newest first
Other references · 17
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
- 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
- 2018nocaps: novel object captioning at scalecited by 444
- 2023BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editingcited by 442
- 2017DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networkscited by 367
- 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
- 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
- 2023ALIP: Adaptive Language-Image Pre-training with Synthetic Captioncited by 70
- 2024CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matchingcited by 61
- 2017Towards Diverse and Natural Image Descriptions via a Conditional GANcited by 59
- 2024Transfer CLIP for Generalizable Image Denoisingcited by 29
5 more
- 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspectivecited by 28
- 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Modelscited by 23
- 2024R-LLaVA: Improving Med-VQA Understanding through Visual Region of Interestcited by 21
- 2024Visual Language Model based Cross-modal Semantic Communication Systemscited by 12
- 2017Speaking the Same Language: Matching Machine to Human Captions by Adversarial Trainingcited by 6
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
- 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
- 2025Test-Time Consistency in Vision Language Models
- 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models
- 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
- 2026Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models
- 2026Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks
- 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Models