# MMErroR: A Benchmark for Erroneous Reasoning in Vision-Language Models

arXiv: https://arxiv.org/abs/2601.03331 · HTML: https://mlsanity.com/paper/2601.03331/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 11 of its 42 references that are in the map.
- First posted on arXiv 6 Jan 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 11 of 11)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning](https://mlsanity.com/paper/2507.01006/) (2025)
- [VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models](https://mlsanity.com/paper/2504.15279/) (2025)
- [FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark](https://mlsanity.com/paper/2509.09680/) (2025)
- [ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detection](https://mlsanity.com/paper/2410.04509/) (2024)
- [VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Models](https://mlsanity.com/paper/2503.07478/) (2025)
- [SEAM: Semantically Equivalent Across Modalities Benchmark for Vision-Language Models](https://mlsanity.com/paper/2508.18179/) (2025)

## Built on by (most cited first, 10 of 10)

- [COMBINER: Composed Image Retrieval Guided by Attribute-based Neighbor Relations](https://mlsanity.com/paper/2606.04604/) (2026)
- [Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs](https://mlsanity.com/paper/2601.22709/) (2026)
- [Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning](https://mlsanity.com/paper/2602.11455/) (2026)
- [TransSplat: Unbalanced Semantic Transport for Language-Driven 3DGS Editing](https://mlsanity.com/paper/2604.19571/) (2026)
- [CoCo-SAM3: Harnessing Concept Conflict in Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2604.19648/) (2026)
- [Modality-Decoupled Online Recursive Editing](https://mlsanity.com/paper/2605.20273/) (2026)
- [IMAGINE: Adaptive Schema-Imagery Enhanced Composition for Composed Video Retrieval](https://mlsanity.com/paper/2606.08144/) (2026)
- [HiP-LoRA: Budgeted Spectral Plasticity for Robust Low-Rank Adaptation](https://mlsanity.com/paper/2604.17751/) (2026)
- [Neutral-Reference Prompting for Vision-Language Models](https://mlsanity.com/paper/2605.15615/) (2026)
- [VEDAL: Variational Error-Driven Asynchronous Learning for 3D Gaussian Splatting Pruning](https://mlsanity.com/paper/2606.02346/) (2026)

## Abstract (arXiv)

Recent advances in Vision-Language Models (VLMs) have improved performance in multi-modal learning, raising the question of whether these models truly understand the content they process. Crucially, can VLMs detect when a reasoning process is wrong and identify its error type? To answer this, we present MMErroR, a multi-modal benchmark of 1997 samples, each embedding a single coherent reasoning error. These samples span 24 subdomains across six top-level domains, ensuring broad coverage and taxonomic richness. Unlike existing benchmarks that focus on answer correctness, MMErroR targets a process-level, error-centric evaluation that requires models to detect incorrect reasoning and classify the error type within both visual and linguistic contexts. We evaluate 12 representative VLMs, and even the best model, Gemini-3-Pro-Preview, classifies the error correctly in only 66.65% of cases, underscoring the challenge of identifying erroneous reasoning. Furthermore, the ability to accurately identify errors offers valuable insights into the capabilities of multi-modal models. Project Page: https://mmerror-benchmark.github.io