mlsanityData to 7 Oct 2026

Edge Reliability Gap in Vision-Language Models: Quantifying Failure Modes of Compressed VLMs Under Visual Corruption

arXiv 2603.26769 ↗First posted 24 Mar 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The rapid compression of large vision-language models (VLMs) for edge deployment raises an underexplored question: do compact models fail differently, not merely more often? This study compares a 7-billion-parameter quantised VLM (Qwen2.5-VL-7B, 4-bit NF4) against a 500-million-parameter FP16 model (SmolVLM2-500M) across 4,000 samples from VQAv2 and COCO Captions. A three-category error taxonomy (Object Blindness, Semantic Drift, Prior Bias) is applied as a diagnostic framework. A text-only GPT-4o judge reveals Semantic Drift (B) as the dominant failure mode on VQAv2 and on COCO for Qwen, with a mixed Object Blindness / Semantic Drift profile for SmolVLM2 on COCO; Prior Bias (C) is present on VQAv2 but absent on COCO for both models. Confidence calibration is measured via Expected Calibration Error (ECE) using geometric mean token probability, compositional reasoning is probed with structured negation probes across four templates, and a blur robustness experiment completes the evaluation. For this model pair, the compact model exhibits a qualitatively distinct failure signature: a 12.5pp larger negation collapse (-33.2pp vs. -20.8pp, Wald 95% CI [8.2, 16.8]pp, p < 10^-8), driven almost entirely by COCO while the VQAv2 gap is not statistically significant (4.5pp, p=0.19). The most discriminating template is false_yn: SMOLVLM2-500M responds "Yes" (incorrectly claiming a depicted object is absent) on 100% of COCO trials vs. 14% for Q WEN 2.5-VL-7B. Asymmetric dataset-dependent miscalibration and a blur experiment with two controlled ablations complete the analysis. The fully reproducible pipeline is released for systematic safety auditing of compressed VLMs prior to edge deployment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 46 of its 56 references that are in the map. First posted on arXiv 24 Mar 2026.

Builds on

46 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Analysing the Robustness of Vision-Language-Models to Common Corruptions18 Apr

Other references · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
33 more
  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  3. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  8. 2018Object Hallucination in Image Captioningcited by 661
  9. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  10. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  11. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Modelscited by 394
  12. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  13. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  14. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  15. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  16. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  17. 2023DepGraph: Towards Any Structural Pruningcited by 202
  18. 2024LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Modelcited by 128
  19. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  20. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Modelscited by 80
  21. 2024Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Modelscited by 51
  22. 2022CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negationcited by 38
  23. 2023Distilling Large Vision-Language Model with Out-of-Distribution Generalizabilitycited by 33
  24. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Modelscited by 31
  25. 2024Toward a Holistic Evaluation of Robustness in CLIP Modelscited by 13
  26. 2025Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbationcited by 11
  27. 2023Distilling Out-of-Distribution Robustness from Vision-Language Foundation Modelscited by 9
  28. 2025Mint: A Simple Test-Time Adaptation of Vision-Language Models against Common Corruptionscited by 5
  29. 2025LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compressioncited by 4
  30. 2023Uncovering the Hidden Cost of Model Compressioncited by 3
  31. 2025Processing and acquisition traces in visual encoders: What does CLIP know about your camera?cited by 3
  32. 2025Less Precise Can Be More Reliable: A Systematic Evaluation of Quantization's Impact on VLMs Beyond Accuracycited by 3
  33. 2024You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
  2. 2025The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs
  3. 2026Hierarchical Compression of Vision-Language Model Benchmarks
  4. 2026Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
  5. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs
  6. 2025Test-Time Consistency in Vision Language Models
  7. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias
  8. 2026Can't Find Waldo: Evaluating VLMs' Sensitivity to Image Resolution and Detail Level