mlsanityData to 14 Oct 2026

MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs

arXiv 2511.14159 ↗First posted 18 Nov 2025cs.CVcited by 15

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Evaluating the robustness of Large Vision-Language Models (LVLMs) is essential for their continued development and responsible deployment in real-world applications. However, existing robustness benchmarks typically focus on hallucination or misleading textual inputs, while largely overlooking the equally critical challenge posed by misleading visual inputs in assessing visual understanding. To fill this important gap, we introduce MVI-Bench, the first comprehensive benchmark specially designed for evaluating how Misleading Visual Inputs undermine the robustness of LVLMs. Grounded in fundamental visual primitives, the design of MVI-Bench centers on three hierarchical levels of misleading visual inputs: Visual Concept, Visual Attribute, and Visual Relationship. Using this taxonomy, we curate six representative categories and compile 1,248 expertly annotated VQA instances. To facilitate fine-grained robustness evaluation, we further introduce MVI-Sensitivity, a novel metric that characterizes LVLM robustness at a granular level. Empirical results across 18 state-of-the-art LVLMs uncover pronounced vulnerabilities to misleading visual inputs, and our in-depth analyses on MVI-Bench provide actionable insights that can guide the development of more reliable and robust LVLMs. The benchmark and codebase can be accessed at https://github.com/chenyil6/MVI-Bench.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 69 of its 78 references that are in the map. First posted on arXiv 18 Nov 2025.

Builds on

69 of 78 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025SAIL-VL2 Technical Report17 Sep
  2. 2025Beyond the Visible: Benchmarking Occlusion Perception in Multimodal Large Language Models6 Aug
  3. 2025Kimi-VL Technical Report10 Apr
  4. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar
  5. 2025Scalable Vision Language Model Training via High Quality Data Curation10 Jan
  6. 2025IllusionBench+: A Large-scale and Comprehensive Benchmark for Visual Illusion Understanding in Vision-Language Models1 Jan
  7. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey26 Nov
  8. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  9. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  10. 2024What matters when building vision-language models?3 May
  11. 2024LVLM-Interpret: An Interpretability Tool for Large Vision-Language Models3 Apr
  12. 2024IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models23 Mar
2 more
  1. 2023How to Configure Good In-Context Sequence for Visual Question Answering4 Dec
  2. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug

Other references · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2025Qwen2.5-VL Technical Reportcited by 5,587
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  12. 2022Emergent Abilities of Large Language Modelscited by 2,512
38 more
  1. 2015VQA: Visual Question Answeringcited by 2,506
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  6. 2023A Survey on Multimodal Large Language Modelscited by 909
  7. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  8. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  9. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  10. 2024Phi-4 Technical Reportcited by 711
  11. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  12. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 388
  13. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  14. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  15. 2025Seedream 4.0: Toward Next-generation Multimodal Image Generationcited by 249
  16. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 248
  17. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  18. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  19. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  20. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  21. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  22. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  23. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
  24. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Modelscited by 86
  25. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 67
  26. 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
  27. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  28. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 43
  29. 2025Emergent Abilities in Large Language Models: A Surveycited by 37
  30. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 30
  31. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Modelscited by 27
  32. 2025More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Modelscited by 27
  33. 2025Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-trainingcited by 23
  34. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  35. 2025Survey of Adversarial Robustness in Multimodal Large Language Modelscited by 16
  36. 2025Mimic In-Context Learning for Multimodal Taskscited by 12
  37. 2024MVTamperBench: Evaluating Robustness of Vision-Language Modelscited by 11
  38. 2025SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materialscited by 8

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Modelscited by 18
  2. 2026Stop When Reasoning Converges: Semantic-Preserving Early Exit for Reasoning Modelscited by 7
  3. 2026ArrowGEV: Grounding Events in Video via Learning the Arrow of Timecited by 4
  4. 2026Rethinking LLM Ensembling from the Perspective of Mixture Modelscited by 1
  5. 2026VisualFLIP: Do Predictions Depend on Task-Critical Visual Evidence in Multimodal Reasoning?cited by 1
  6. 2026HoloCount: A Holistic Visual Counting Benchmark for MLLMscited by 1
  7. 2026Adaptive Multi-Stage Patent Claim Generation with Unified Quality Assessmentcited by 0
  8. 2026ProHiFlo: Hierarchical Flow Matching with Functional Guidance for De Novo Protein Generationcited by 0
  9. 2026FinInvest-GTCN: Explainable Graph-Temporal-Causal Modeling for Risk-Aware Investment Decision Optimizationcited by 0
  10. 2026EVLA: An Electro-Aware Multimodal Assistant for Physically-Grounded Driving Reasoning and Controlcited by 0
  11. 2026A3M: Adaptive, Adversarial and Multi-Objective Learning for Strategic Bidding in Repeated Auctionscited by 0
  12. 2026ProWAFT: A ROMA-LPD Instance for Workload-Aware and Dynamic Fault Tolerance in FPGA-Based CNN Acceleratorscited by 0
3 more
  1. 2026Beyond What Meets the Eye: Unveiling Situational Illusions for Multimodal Large Language Modelscited by 0
  2. 2026Graph-Based Modeling of Financial Volatility Dynamicscited by 0
  3. 2026PULSE: Identifying Demonstration-Utility Features with Sparse Autoencoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unveiling the Lack of LVLM Robustness to Fundamental Visual Variations: Why and Path Forward
  2. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models
  3. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  4. 2026Validating, Not Sampling: Region-Level Robustness of Vision-Language and Vision-Language-Action Models
  5. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?
  6. 2026Scaling Test-Time Robustness of Vision-Language Models via Self-Critical Inference Framework
  7. 2026SynDORBench: Evaluating LVLM Perceptual Robustness Under Physically Constrained Visibility Conditions
  8. 2026Scaling Vision-Language Models Is Not Enough to Mitigate Bias