mlsanityData to 7 Oct 2026

Beyond Standard Benchmarks: A Systematic Audit of Vision-Language Model's Robustness to Natural Semantic Variation Across Diverse Tasks

arXiv 2604.04473 ↗First posted 6 Apr 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

Abstract (arXiv)

Recent advances in vision-language models (VLMs) trained on web-scale image-text pairs have enabled impressive zero-shot transfer across a diverse range of visual tasks. However, comprehensive and independent evaluation beyond standard benchmarks is essential to understand their robustness, limitations, and real-world applicability. This paper presents a systematic evaluation framework for VLMs under natural adversarial scenarios for diverse downstream tasks, which has been overlooked in previous evaluation works. We evaluate a wide range of VLMs (CLIP, robust CLIP, BLIP2, and SigLIP2) on curated adversarial datasets (typographic attacks, ImageNet-A, and natural language-induced adversarial examples). We measure the natural adversarial performance of selected VLMs for zero-shot image classification, semantic segmentation, and visual question answering. Our analysis reveals that robust CLIP models can amplify natural adversarial vulnerabilities, and CLIP models significantly reduce performance for natural language-induced adversarial examples. Additionally, we provide interpretable analyses to identify failure modes. We hope our findings inspire future research in robust and fair multimodal pattern recognition.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 38 references that are in the map. First posted on arXiv 6 Apr 2026.

Builds on

33 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models3 Jan
  2. 2024Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models19 Feb
  3. 2023How Robust is Google's Bard to Adversarial Image Attacks?21 Sep
  4. 2023On Evaluating Adversarial Robustness of Large Vision-Language Models26 May
  5. 2023Defense-Prefix for Preventing Typographic Attacks on CLIP10 Apr

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  8. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  9. 2019Natural Adversarial Examplescited by 1,401
  10. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
  11. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  12. 2025SAM 3: Segment Anything with Conceptscited by 835
16 more
  1. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  2. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  3. 2021Image Segmentation Using Text and Image Promptscited by 509
  4. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  5. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  6. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 316
  7. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  8. 2022Patching open-vocabulary models by interpolating weightscited by 187
  9. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
  10. 2024Sparse autoencoders reveal selective remapping of visual concepts during adaptationcited by 66
  11. 2022Disentangling visual and written concepts in CLIPcited by 57
  12. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  13. 2025On the Limitations of Vision-Language Models in Understanding Image Transformscited by 12
  14. 2022Enhancing Adversarial Robustness for Deep Metric Learningcited by 10
  15. 2024Natural Language Induced Adversarial Imagescited by 4
  16. 2025On the Brittleness of CLIP Text Encoderscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting the Adversarial Robustness of Vision Language Models: a Multimodal Perspective
  2. 2025Navigating the Trade-off: A Synthesis of Defensive Strategies for Zero-Shot Adversarial Robustness in Vision-Language Models
  3. 2025Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness
  4. 2024Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques
  5. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditions
  6. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples
  7. 2026Robustifying Vision-Language Models via Test-Time Prompt Adaptation
  8. 2024A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models