mlsanityData to 14 Oct 2026

Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark

arXiv 2506.04280 ↗First posted 4 Jun 2025cs.CV cs.AIcited by 21

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

With enhanced capabilities and widespread applications, Multimodal Large Language Models (MLLMs) are increasingly required to process and reason over multiple images simultaneously. However, existing MLLM benchmarks focus either on single-image visual reasoning or on multi-image understanding tasks with only final-answer evaluation, leaving the reasoning capabilities of MLLMs over multi-image inputs largely underexplored. To address this gap, we introduce the Multimodal Multi-image Reasoning Benchmark (MMRB), the first benchmark designed to evaluate structured visual reasoning across multiple images. MMRB comprises 92 sub-tasks covering spatial, temporal, and semantic reasoning, with multi-solution, CoT-style annotations generated by GPT-4o and refined by human experts. A derivative subset is designed to evaluate multimodal reward models in multi-image scenarios. To support fast and scalable evaluation, we propose a sentence-level matching framework using open-source LLMs. Extensive baseline experiments on 40 MLLMs, including 9 reasoning-specific models and 8 reward models, demonstrate that open-source MLLMs still lag significantly behind commercial MLLMs in multi-image reasoning tasks. Furthermore, current multimodal reward models are nearly incapable of handling multi-image reward ranking tasks.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 50 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

33 of 50 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models21 Apr
  2. 2025Benchmarking Multimodal CoT Reward Model Stepwise by Visual Program9 Apr
  3. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoning13 Mar
  4. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency13 Feb
  5. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model21 Jan
  6. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs10 Jan
  7. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  8. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  9. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  10. 2024MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps18 Oct
  11. 2024M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought26 May
  12. 2024WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoning6 May
2 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  2. 2023Multimodal Chain-of-Thought Reasoning in Language Models2 Feb

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  5. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  8. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  9. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  10. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  11. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  12. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
7 more
  1. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  2. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 77
  3. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  4. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  5. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 44
  6. 2019NLVR2 Visual Bias Analysiscited by 19
  7. 2018Generalizing Across Multi-Objective Reward Functions in Deep Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 15.

21 papers.

Sort:

By area:

  1. 2025AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoningcited by 20
  2. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoningcited by 9
  3. 2026OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Modelcited by 5
  4. 2025MATRIX: Mask Track Alignment for Interaction-aware Video Generationcited by 4
  5. 2026TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoningcited by 4
  6. 2026IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Productscited by 3
  7. 2025Multimodal LLMs See Sentimentcited by 2
  8. 2026Multimodal Language Models Cannot Spot Spatial Inconsistenciescited by 2
  9. 2025MIST: Towards Multi-dimensional Implicit BiaS Evaluation of LLMs for Theory of Mindcited by 1
  10. 2025Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretationcited by 1
  11. 2026Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judgecited by 1
  12. 2026CORAL: Correspondence Alignment for Improved Virtual Try-Oncited by 1
9 more
  1. 2025V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questionscited by 0
  2. 2026DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modelingcited by 0
  3. 2026StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoningcited by 0
  4. 2026MMClima: A Framework for Multimodal Climate Science Data and Evaluationcited by 0
  5. 2026CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Modelscited by 0
  6. 2026OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understandingcited by 0
  7. 2026Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessmentcited by 0
  8. 2026Automatic multimodal UX improvement recommendations from LLM agent user simulationscited by 0
  9. 2026MVGraph: Benchmarking LVLMs on Evidence Composition across Multi-View Visual Graphcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  2. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
  3. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  4. 2025From Easy to Hard: The MIR Benchmark for Progressive Interleaved Multi-Image Reasoning
  5. 2023Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Models
  6. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  7. 2025OCR-Reasoning Benchmark: Unveiling the True Capabilities of MLLMs in Complex Text-Rich Image Reasoning
  8. 2024MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs