mlsanityData to 7 Oct 2026

VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models

arXiv 2603.06148 ↗First posted 6 Mar 2026cs.CV cs.AIcited by 12

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) achieve strong performance on standard, high-quality datasets, but we still do not fully understand how they perform under real-world image distortions. We present VLM-RobustBench, a benchmark spanning 49 augmentation types across noise, blur, weather, digital, and geometric perturbations, evaluated under graded severities (low/mid/high) and binary transforms, yielding 133 corrupted settings. We evaluate VLMs from four families (Qwen, InternVL, Molmo, Gemma) on two complementary benchmarks: MMBench (visually grounded) and MMMU-Pro (reasoning-oriented). Our results reveal that visual severity is a weak predictor of difficulty: low-severity spatial perturbations often degrade performance more than visually severe photometric corruptions. In particular, low-severity glass_blur reduces MMBench accuracy by about 8 pp on average across models, while the largest drops arise from resampling and geometric distortions (e.g., upsample, elastic_transform), reaching up to 34 pp. Overall, our findings suggest current VLMs are semantically strong but spatially fragile, motivating the definition of novel robustness evaluation protocols and training regimes that emphasize resampling and geometric invariances.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 29 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

23 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly15 May
  2. 2025Analysing the Robustness of Vision-Language-Models to Common Corruptions18 Apr
  3. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples18 Oct

Other references · 20

  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  5. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  6. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  7. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  8. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  9. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 307
  10. 2023Are aligned neural networks adversarially aligned?cited by 305
  11. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 281
  12. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
8 more
  1. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  2. 2021Vision Transformers are Robust Learnerscited by 208
  3. 2022Understanding The Robustness in Vision Transformerscited by 144
  4. 2022Understanding Zero-Shot Adversarial Robustness for Large-Scale Modelscited by 117
  5. 2025VLA-0: Building State-of-the-Art VLAs with Zero Modificationcited by 49
  6. 2024RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learningcited by 45
  7. 2024A Survey on Evaluation of Out-of-Distribution Generalizationcited by 20
  8. 2025On the Perception Bottleneck of VLMs for Chart Understandingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026RemoteShield: Enable Robust Multimodal Large Language Models for Earth Observationcited by 2
  2. 2026RoboStressBench: Benchmarking VLM Robustness to Physical Visual Stress in Embodied Scenescited by 2
  3. 2026Are Reasoning Vision-Language Models Robust to Semantic Visual Distractions?cited by 2
  4. 2026Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learningcited by 1
  5. 2026Revealing Interpretable Failure Modes of VLMscited by 0
  6. 2026TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videoscited by 0
  7. 2026SpaceDG: Benchmarking Spatial Intelligence under Visual Degradationcited by 0
  8. 2026How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbationscited by 0
  9. 2026The Illusion of Replacement: Rethinking Specialized Machine Learning Models in the Foundation Model Eracited by 0
  10. 2026Assessing Adversarial Robustness of Latent Reasoning Modelscited by 0
  11. 2026Reading Right, Answering Wrong: How Visual Configuration Changes Affect Evidence Use in VLMscited by 0
  12. 2026Can't Find Waldo: Evaluating VLMs' Sensitivity to Image Resolution and Detail Levelcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models
  2. 2025Evaluating Robustness of Vision-Language Models Under Noisy Conditions
  3. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs
  4. 2025On the Domain Robustness of Contrastive Vision-Language Models
  5. 2025Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
  6. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  7. 2026DistortBench: Benchmarking Vision Language Models on Image Distortion Identification
  8. 2026PRISM-VLM: A Multi-Axis Discriminative Benchmark for Compact Vision-Language Models