mlsanityData to 7 Oct 2026

Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metric

arXiv 2605.06201 ↗First posted 7 May 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Dominant accuracy evaluation might reward unwarranted guessing of Large Language Models, and it might not be applicable to novel tasks for model validation without ground-truth (gt) annotation. Based on basic logic principle, we propose a novel framework to evaluate the vision-language logical consistency of MLLMs on both sufficient and necessary cause-effect relations. We define Vision-Language Logical Consistency Metric (VL-LCM) on traditional MC-VQA tests, and recent NaturalBench tests without the need for gt annotation. Through systematic experiments on representative VL benchmark MMMU and recent VL challenges like NaturalBench, we evaluated 11 recent open-source MLLMs from 4 frontier families. Our findings reveal that, despite significant progress of recent MLLMs on accuracy, logical consistency lags behind significantly. Extensive evaluations on the correlations of VL-LCM with metrics on gt, the reliability of LCM, and the relation of VL-LCM with response distribution justify the validity and applicability of VL-LCM even without gt annotation. Our findings suggest that, beyond accuracy, logical consistency could be employed for both accuracy and reliability. VL-LCM can also be employed for MLLM selection, validation, and reliable answer justification in novel tasks without gt annotation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 45 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

36 of 45 references.

References · 36

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  7. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2025Gemma 3 Technical Reportcited by 1,686
  10. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  11. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
24 more
  1. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  2. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  3. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 331
  4. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  5. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  6. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  7. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  8. 2025Vision-Language Models Do Not Understand Negationcited by 69
  9. 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directionscited by 66
  10. 2019A Logic-Driven Framework for Consistency of Neural Modelscited by 64
  11. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  12. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  13. 2022Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inferencecited by 40
  14. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 31
  15. 2024Unveiling the Tapestry of Consistency in Large Vision-Language Modelscited by 22
  16. 2023A Closer Look into Automatic Evaluation Using Large Language Modelscited by 17
  17. 2024Logical Consistency of Large Language Models in Fact-checkingcited by 15
  18. 2025Promptception: How Sensitive Are Large Multimodal Models to Prompts?cited by 13
  19. 2022CARETS: A Consistency And Robustness Evaluative Test Suite for VQAcited by 11
  20. 2024MM-R3: On (In-)Consistency of Vision-Language Models (VLMs)cited by 10
  21. 2025The Future of MLLM Prompting is Adaptive: A Comprehensive Experimental Evaluation of Prompt Engineering Methods for Robust Multimodal Performancecited by 7
  22. 2023Logical Implications for Visual Question Answering Consistencycited by 6
  23. 2025Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilizationcited by 4
  24. 2024Towards Logically Consistent Language Models via Probabilistic Reasoningcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Test-Time Consistency in Vision Language Models
  2. 2026JAMMEval: A Refined Collection of Japanese Benchmarks for Reliable VLM Evaluation
  3. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models
  4. 2026Explicit Logic Channel for Validation and Enhancement of MLLMs on Zero-Shot Tasks
  5. 2025Can Argus Judge Them All? Comparing VLMs Across Domains
  6. 2026MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models
  7. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMs
  8. 2024Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison