# Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark arXiv: https://arxiv.org/abs/2506.04280 ยท HTML: https://mlsanity.com/paper/2506.04280/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. - Builds on 33 of its 50 references that are in the map. - First posted on arXiv 4 Jun 2025. ## Placed in - LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/) - Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/) - LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/) ## Builds on (most cited first, 30 of 33) - [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017) - [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024) - [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025) - [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023) - [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024) - [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024) - [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023) - [Multimodal Chain-of-Thought Reasoning in Language Models](https://mlsanity.com/paper/2302.00923/) (2023) - [Generative Verifiers: Reward Modeling as Next-Token Prediction](https://mlsanity.com/paper/2408.15240/) (2024) - [LLaVA-CoT: Let Vision Language Models Reason Step-by-Step](https://mlsanity.com/paper/2411.10440/) (2024) - [Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision](https://mlsanity.com/paper/2309.14181/) (2023) - [Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization](https://mlsanity.com/paper/2411.10442/) (2024) - [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024) - [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://mlsanity.com/paper/2406.09411/) (2024) - [LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs](https://mlsanity.com/paper/2501.06186/) (2025) - [VisualPRM: An Effective Process Reward Model for Multimodal Reasoning](https://mlsanity.com/paper/2503.10291/) (2025) - [LLaVA-Critic: Learning to Evaluate Multimodal Models](https://mlsanity.com/paper/2410.02712/) (2024) - [MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency](https://mlsanity.com/paper/2502.09621/) (2025) - [M3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought](https://mlsanity.com/paper/2405.16473/) (2024) - [VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models](https://mlsanity.com/paper/2504.15279/) (2025) - [MileBench: Benchmarking MLLMs in Long Context](https://mlsanity.com/paper/2404.18532/) (2024) - [Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions](https://mlsanity.com/paper/2308.04152/) (2023) - [InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model](https://mlsanity.com/paper/2501.12368/) (2025) - [MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models](https://mlsanity.com/paper/2408.02718/) (2024) - [R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning](https://mlsanity.com/paper/2505.02835/) (2025) - [Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Models](https://mlsanity.com/paper/2502.14191/) (2025) - [NLVR2 Visual Bias Analysis](https://mlsanity.com/paper/1909.10411/) (2019) - [WorldQA: Multimodal World Knowledge in Videos through Long-Chain Reasoning](https://mlsanity.com/paper/2405.03272/) (2024) ## Built on by (most cited first, 21 of 21) - [AirCopBench: A Benchmark for Multi-drone Collaborative Embodied Perception and Reasoning](https://mlsanity.com/paper/2511.11025/) (2025) - [MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning](https://mlsanity.com/paper/2603.02024/) (2026) - [OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model](https://mlsanity.com/paper/2604.20806/) (2026) - [MATRIX: Mask Track Alignment for Interaction-aware Video Generation](https://mlsanity.com/paper/2510.07310/) (2025) - [TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoning](https://mlsanity.com/paper/2601.16520/) (2026) - [IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products](https://mlsanity.com/paper/2606.14383/) (2026) - [Multimodal LLMs See Sentiment](https://mlsanity.com/paper/2508.16873/) (2025) - [Multimodal Language Models Cannot Spot Spatial Inconsistencies](https://mlsanity.com/paper/2604.00799/) (2026) - [MIST: Towards Multi-dimensional Implicit BiaS Evaluation of LLMs for Theory of Mind](https://mlsanity.com/paper/2506.14161/) (2025) - [Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretation](https://mlsanity.com/paper/2510.12953/) (2025) - [Bi-Level Prompt Optimization for Multimodal LLM-as-a-Judge](https://mlsanity.com/paper/2602.11340/) (2026) - [CORAL: Correspondence Alignment for Improved Virtual Try-On](https://mlsanity.com/paper/2602.17636/) (2026) - [V-REX: Benchmarking Exploratory Visual Reasoning via Chain-of-Questions](https://mlsanity.com/paper/2512.11995/) (2025) - [DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modeling](https://mlsanity.com/paper/2604.19544/) (2026) - [StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoning](https://mlsanity.com/paper/2606.00148/) (2026) - [MMClima: A Framework for Multimodal Climate Science Data and Evaluation](https://mlsanity.com/paper/2606.10194/) (2026) - [CoRe: A Comprehensive Framework for Cross-Image Comparative Reasoning in Vision-Language Models](https://mlsanity.com/paper/2607.12786/) (2026) - [OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding](https://mlsanity.com/paper/2608.30678/) (2026) - [Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment](https://mlsanity.com/paper/2609.02573/) (2026) - [Automatic multimodal UX improvement recommendations from LLM agent user simulations](https://mlsanity.com/paper/2609.22971/) (2026) - [MVGraph: Benchmarking LVLMs on Evidence Composition across Multi-View Visual Graph](https://mlsanity.com/paper/2610.12961/) (2026) ## Abstract (arXiv) With enhanced capabilities and widespread applications, Multimodal Large Language Models (MLLMs) are increasingly required to process and reason over multiple images simultaneously. However, existing MLLM benchmarks focus either on single-image visual reasoning or on multi-image understanding tasks with only final-answer evaluation, leaving the reasoning capabilities of MLLMs over multi-image inputs largely underexplored. To address this gap, we introduce the Multimodal Multi-image Reasoning Benchmark (MMRB), the first benchmark designed to evaluate structured visual reasoning across multiple images. MMRB comprises 92 sub-tasks covering spatial, temporal, and semantic reasoning, with multi-solution, CoT-style annotations generated by GPT-4o and refined by human experts. A derivative subset is designed to evaluate multimodal reward models in multi-image scenarios. To support fast and scalable evaluation, we propose a sentence-level matching framework using open-source LLMs. Extensive baseline experiments on 40 MLLMs, including 9 reasoning-specific models and 8 reward models, demonstrate that open-source MLLMs still lag significantly behind commercial MLLMs in multi-image reasoning tasks. Furthermore, current multimodal reward models are nearly incapable of handling multi-image reward ranking tasks.