# Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models

arXiv: https://arxiv.org/abs/2502.16033 · HTML: https://mlsanity.com/paper/2502.16033/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 22 Feb 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://mlsanity.com/paper/2112.08542/) (2021)
- [CogCoM: A Visual Language Model with Chain-of-Manipulations Reasoning](https://mlsanity.com/paper/2402.04236/) (2024)
- [ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshots](https://mlsanity.com/paper/2209.08199/) (2022)
- [Harnessing Webpage UIs for Text-Rich Visual Understanding](https://mlsanity.com/paper/2410.13824/) (2024)
- [Fast and Accurate Factual Inconsistency Detection Over Long Documents](https://mlsanity.com/paper/2310.13189/) (2023)
- [GPT-4 as an Effective Zero-Shot Evaluator for Scientific Figure Captions](https://mlsanity.com/paper/2310.15405/) (2023)
- [Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding](https://mlsanity.com/paper/2406.19263/) (2024)

## Built on by (most cited first, 24 of 24)

- [How Do Vision-Language Models Process Conflicting Information Across Modalities?](https://mlsanity.com/paper/2507.01790/) (2025)
- [Mind with Eyes: from Language Reasoning to Multimodal Reasoning](https://mlsanity.com/paper/2503.18071/) (2025)
- [When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs](https://mlsanity.com/paper/2511.02243/) (2025)
- [Reinforcement Learning for Large Model: A Survey](https://mlsanity.com/paper/2508.08189/) (2025)
- [Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs](https://mlsanity.com/paper/2512.08923/) (2025)
- [M3CoTBench: Benchmark Chain-of-Thought of MLLMs in Medical Image Understanding](https://mlsanity.com/paper/2601.08758/) (2026)
- [Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstention](https://mlsanity.com/paper/2603.27187/) (2026)
- [Mechanisms of Prompt-Induced Hallucination in Vision-Language Models](https://mlsanity.com/paper/2601.05201/) (2026)
- [PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies](https://mlsanity.com/paper/2510.16505/) (2025)
- [Early Estimation of Language to Latent Alignment in Diffusion Models](https://mlsanity.com/paper/2512.08505/) (2025)
- [MARBLE: A Hard Benchmark for Multimodal Spatial Reasoning and Planning](https://mlsanity.com/paper/2506.22992/) (2025)
- [Can Large Multimodal Models Actively Recognize Faulty Inputs? A Systematic Evaluation Framework of Their Input Scrutiny Ability](https://mlsanity.com/paper/2508.04017/) (2025)
- [CAVE: Detecting and Explaining Commonsense Anomalies in Visual Environments](https://mlsanity.com/paper/2510.26006/) (2025)
- [CLASH: A Benchmark for Cross-Modal Contradiction Detection](https://mlsanity.com/paper/2511.19199/) (2025)
- [MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments](https://mlsanity.com/paper/2604.13418/) (2026)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [What Color Is the Text? A Benchmark for Hallucination Induced by Image-Embedded Prompt](https://mlsanity.com/paper/2511.13400/) (2025)
- [Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning](https://mlsanity.com/paper/2602.23351/) (2026)
- [Benchmarking Vision-Language Models under Contradictory Virtual Content Attacks in Augmented Reality](https://mlsanity.com/paper/2604.05510/) (2026)
- [Hidden in Plain Sight: Reasoning in Underspecified and Misspecified Scenarios for Multimodal LLMs](https://mlsanity.com/paper/2506.00258/) (2025)
- [Struct-Searcher: Agentic Structural Thinking Advances Multimodal Deep Information Seeking](https://mlsanity.com/paper/2606.07689/) (2026)
- [ArtiFact: A Large-Scale Multi-Modal Cultural Heritage Dataset](https://mlsanity.com/paper/2606.09648/) (2026)
- [Prior Directions: Why GUI Grounding Gets Locked in the Past](https://mlsanity.com/paper/2607.26913/) (2026)
- [Re2A: Situated Conversational Recommendation via Rubric-based Preference Reasoning and Alignment](https://mlsanity.com/paper/2609.18249/) (2026)

## Abstract (arXiv)

Existing Multimodal Large Language Models (MLLMs) are predominantly trained and tested on consistent visual-textual inputs, leaving open the question of whether they can handle inconsistencies in real-world, layout-rich content. To bridge this gap, we propose the Multimodal Inconsistency Reasoning (MMIR) benchmark to assess MLLMs' ability to detect and reason about semantic mismatches in artifacts such as webpages, presentation slides, and posters. MMIR comprises 534 challenging samples, each containing synthetically injected errors across five reasoning-heavy categories: Factual Contradiction, Identity Misattribution, Contextual Mismatch, Quantitative Discrepancy, and Temporal/Spatial Incoherence. We evaluate six state-of-the-art MLLMs, showing that models with dedicated multimodal reasoning capabilities, such as o1, substantially outperform their counterparts while open-source models remain particularly vulnerable to inconsistency errors. Detailed error analyses further show that models excel in detecting pairwise inconsistencies but struggle with inconsistencies confined to single elements in complex layouts. Probing experiments reveal that single-modality prompting, including Chain-of-Thought (CoT) and Set-of-Mark (SoM) methods, yields marginal gains, revealing a key bottleneck in cross-modal reasoning. Our findings highlight the need for advanced multimodal reasoning and point to future research on multimodal inconsistency.