mlsanityData to 14 Oct 2026

Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability

arXiv 2508.04017 ↗First posted 6 Aug 2025cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large Multimodal Models (LMMs) have witnessed remarkable growth, showcasing formidable capabilities in handling intricate multimodal tasks with exceptional performance. Recent research has underscored the inclination of large language models to passively accept defective inputs, often resulting in futile reasoning on invalid prompts. However, the same critical question of whether LMMs can actively detect and scrutinize erroneous inputs still remains unexplored. To address this gap, we introduce the Input Scrutiny Ability Evaluation Framework (ISEval), which encompasses seven categories of flawed premises and three evaluation metrics. Our extensive evaluation of ten advanced LMMs has identified key findings. Most models struggle to actively detect flawed textual premises without guidance, which reflects a strong reliance on explicit prompts for premise error identification. Error type affects performance: models excel at identifying logical fallacies but struggle with surface-level linguistic errors and certain conditional flaws. Modality trust varies-Gemini 2.5 pro and Claude Sonnet 4 balance visual and textual info, while aya-vision-8b over-rely on text in conflicts. These insights underscore the urgent need to enhance LMMs' proactive verification of input validity and shed novel insights into mitigating the problem. The code is available at https://github.com/MLGroupJLU/LMM_ISEval.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 38 references that are in the map. First posted on arXiv 6 Aug 2025.

Builds on

29 of 38 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Model27 May
  2. 2025MLLMs are Deeply Affected by Modality Bias24 May
  3. 2025Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models22 Feb
  4. 2024Surveying the MLLM Landscape: A Meta-Review of Current Surveys17 Sep
  5. 2024Dissecting Dissonance: Benchmarking Large Multimodal Models Against Self-Contradictory Instructions2 Aug

Other references · 24

  1. 2025Qwen2.5-VL Technical Reportcited by 5,587
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  4. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  5. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?cited by 89
  6. 2024CriticBench: Benchmarking LLMs for Critique-Correct Reasoningcited by 88
  7. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
  8. 2025Missing Premise exacerbates Overthinking: Are Reasoning Models losing Critical Thinking Skill?cited by 58
  9. 2024Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correctioncited by 45
  10. 2025Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?cited by 44
  11. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  12. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
12 more
  1. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Modelscited by 30
  2. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  3. 2024Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Imagecited by 26
  4. 2025Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Modelscited by 23
  5. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?cited by 23
  6. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decodingcited by 22
  7. 2023Critique Ability of Large Language Modelscited by 20
  8. 2025Let's Verify Math Questions Step by Stepcited by 18
  9. 2025Robust Multimodal Large Language Models Against Modality Conflictcited by 18
  10. 2025On the robustness of multimodal language model towards distractionscited by 9
  11. 2025Don't Take the Premise for Granted: Evaluating the Premise Critique Ability of Large Language Modelscited by 8
  12. 2025MathDebugger: Detecting and Diagnosing Errors in Synthetic Mathematical Datacited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistenciescited by 4
  2. 2025Judge Before Answer: Can MLLM Discern the False Premise in Question?cited by 2
  3. 2026MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hidden in Plain Sight: Reasoning in Underspecified and Misspecified Scenarios for Multimodal LLMs
  2. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  3. 2025MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
  4. 2026Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs
  5. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning
  6. 2026Same Semantics, Different Outcome: On the Modality Robustness of Multimodal LLMs under Knowledge Conflict
  7. 2024Understanding the Role of LLMs in Multimodal Evaluation Benchmarks
  8. 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning