mlsanityData to 14 Oct 2026

Diagnosing Corruption-Induced Reliability Failures in Vision-Language Models

arXiv 2511.19032 ↗First posted 24 Nov 2025cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual corruptions can change vision--language model (VLM) behavior in ways that top-1 accuracy does not capture. A model may keep the same answer while losing distributional support, or improve accuracy through unstable wrong-to-correct changes. We introduce Bench-C, a controlled multiple-choice testbed for studying these effects. It selects semantically diverse samples whose predictions respond to corruption, and evaluates them under 19 corruption types and five severity levels. To measure how corruption changes the option distribution, we introduce the Robustness Alignment Score (RAS), which combines confidence-correctness alignment with uncertainty direction. We further separate originally correct samples from originally wrong samples, and track whether changes are temporary or persistent across severity. Experiments across 13 VLMs reveal a counterintuitive pattern: mild corruptions can improve top-1 accuracy while degrading prediction structure. These failures include silent degradation, erroneous overconfidence, and severity-dependent persistence. Bench-C therefore supports robustness evaluation that goes beyond final answers and attributes where reliability changes occur. Code and data are available at https://github.com/xiangjieSui/Bench-C.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 51 references that are in the map. First posted on arXiv 24 Nov 2025.

Builds on

40 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Analysing the Robustness of Vision-Language-Models to Common Corruptions18 Apr
  2. 2025Kimi-VL Technical Report10 Apr
  3. 2025Scalable Vision Language Model Training via High Quality Data Curation10 Jan
  4. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  5. 2024Building and better understanding vision-language models: insights and future directions22 Aug
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  7. 2023Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations2 Oct

Other references · 33

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2017On Calibration of Modern Neural Networkscited by 4,960
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2025Gemma 3 Technical Reportcited by 1,824
  9. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  10. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  11. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  12. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
21 more
  1. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  2. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 292
  3. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  4. 2023Proving Test Set Contamination in Black Box Language Modelscited by 230
  5. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 200
  6. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  7. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  8. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  9. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  10. 2025LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?cited by 49
  11. 2023Sibling-Attack: Rethinking Transferable Adversarial Attacks against Face Recognitioncited by 34
  12. 2024R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?cited by 30
  13. 2025Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcutscited by 29
  14. 20242AFC Prompting of Large Multimodal Models for Image Quality Assessmentcited by 20
  15. 2023The Best Defense is a Good Offense: Adversarial Augmentation against Adversarial Attackscited by 15
  16. 2024Falcon2-11B Technical Reportcited by 12
  17. 2024MVTamperBench: Evaluating Robustness of Vision-Language Modelscited by 11
  18. 2024Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustnesscited by 10
  19. 2022Tracing the Origin of Adversarial Attack for Forensic Investigation and Deterrencecited by 3
  20. 2025Demystifying the Visual Quality Paradox in Multimodal Large Language Modelscited by 3
  21. 2024Improving Adversarial Robustness via Decoupled Visual Representation Maskingcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learningcited by 1
  2. 2026MedQ-Deg: A Multidimensional Benchmark for Evaluating MLLMs Across Medical Image Quality Degradationscited by 0
  3. 2026BRUCE: Benchmarking Robustness Under Corruption Escalation for Scientific Vision-Language Reasoningcited by 0
  4. 2026What Paired Evaluations Reveal under Visual Perturbationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pathological Truth Bias in Vision-Language Models
  2. 2026Small Vision-Language Models Know When They Are Wrong But Cannot Say So: A Two-Model Study of Stated versus Internal Confidence Under Realistic Image Degradation
  3. 2025Know What You do Not Know: Verbalized Uncertainty Estimation Robustness on Corrupted Images in Vision-Language Models
  4. 2026Bigger or Cheaper? Scale and Quantization Effects on Uncertainty Signals in Vision-Language Models Under Image Degradation
  5. 2026Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption
  6. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
  7. 2026Towards Reliable Vision-Language Models for Autonomous Driving
  8. 2026How Robust is OCR-Reasoning? Evaluating OCR-Reasoning Robustness of Vision-Language Models under Visual Perturbations