mlsanityData to 7 Oct 2026

Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning

arXiv 2406.12742 ↗First posted 18 Jun 2024cs.CV cs.AI cs.CLcited by 50

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

The advancement of large language models (LLMs) has significantly broadened the scope of applications in natural language processing, with multi-modal LLMs extending these capabilities to integrate and interpret visual data. However, existing benchmarks for visual language models (VLMs) predominantly focus on single-image inputs, neglecting the crucial aspect of multi-image understanding. In this paper, we introduce a Multi-Image Relational Benchmark MIRB, designed to evaluate VLMs' ability to compare, analyze, and reason across multiple images. Our benchmark encompasses four categories: perception, visual world knowledge, reasoning, and multi-hop reasoning. Through a comprehensive evaluation of a wide range of open-source and closed-source models, we demonstrate that while open-source VLMs were shown to approach the performance of GPT-4V in single-image tasks, a significant performance gap remains in multi-image reasoning tasks. Our findings also reveal that even the state-of-the-art GPT-4V model struggles with our benchmark, underscoring the need for further research and development in this area. We believe our contribution of MIRB could serve as a testbed for developing the next-generation multi-modal models.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 49 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

26 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models22 Mar
  2. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  3. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs27 Nov
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 22

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  11. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  12. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
10 more
  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  2. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  3. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  4. 2024What matters when building vision-language models?cited by 321
  5. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  6. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  8. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 63
  9. 2023Fool Your (Vision and) Language Model With Embarrassingly Simple Permutationscited by 24
  10. 2023Vision Learners Meet Web Image-Text Pairscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 27; 2026: 14.

50 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  5. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  6. 2025MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agentscited by 35
  7. 2025VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cuescited by 33
  8. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generationcited by 30
  9. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 24
  10. 2024Leopard: A Vision Language Model For Text-Rich Multi-Image Taskscited by 18
  11. 2024By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Promptingcited by 12
  12. 2025Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chainscited by 11
38 more
  1. 2025PAL-UI: Planning with Active Look-back for Vision-Based GUI Agentscited by 11
  2. 2024MMRA: A Benchmark for Evaluating Multi-Granularity and Multi-Image Relational Association Capabilities in Large Visual Language Modelscited by 9
  3. 2025MiCo: Multi-image Contrast for Reinforcement Visual Reasoningcited by 9
  4. 2025Exploring and Evaluating Multimodal Knowledge Reasoning Consistency of Multimodal Large Language Modelscited by 8
  5. 2025VReST: Enhancing Reasoning in Large Vision-Language Models through Tree Search and Self-Reward Mechanismcited by 7
  6. 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modescited by 7
  7. 2024MineAgent: Towards Remote-Sensing Mineral Exploration with Multimodal Large Language Modelscited by 6
  8. 2025VisEscape: A Benchmark for Evaluating Exploration-driven Decision-making in Virtual Escape Roomscited by 6
  9. 2026Where Does the Answer Come From? Benchmarking View-Level Visual Evidence Identification in Multi-View MLLMs for Autonomous Drivingcited by 6
  10. 2025Enhancing Vision-Language Model Training with Reinforcement Learning in Synthetic Worlds for Real-World Successcited by 4
  11. 2025RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answeringcited by 4
  12. 2025Do MLLMs Capture How Interfaces Guide User Behavior? A Benchmark for Multimodal UI/UX Design Understandingcited by 3
  13. 2025ChartAB: A Benchmark for Chart Grounding & Dense Alignmentcited by 3
  14. 2025VisChainBench: A Benchmark for Multi-Turn, Multi-Image Visual Reasoning Beyond Language Priorscited by 3
  15. 2026Pix2Fact: When Vision Is Not Enough -- Benchmarking Fine-Grained VQA with Web Verification on High-Resolution Real-World Scenescited by 3
  16. 2026OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Modelcited by 3
  17. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lenscited by 2
  18. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  19. 2025See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosiscited by 2
  20. 2025MIHBench: Benchmarking and Mitigating Multi-Image Hallucinations in Multimodal Large Language Modelscited by 2
  21. 2026Looking Back and Forth: Cross-Image Attention Calibration and Attentive Preference Learning for Multi-Image Hallucination Mitigationcited by 2
  22. 2024CAST: Cross-modal Alignment Similarity Test for Vision Language Modelscited by 1
  23. 2025Can Large Language Models Unveil the Mysteries? An Exploration of Their Ability to Unlock Information in Complex Scenarioscited by 1
  24. 2025FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Modelscited by 1
  25. 2025Analyze-Prompt-Reason: A Collaborative Agent-Based Framework for Multi-Image Vision-Language Reasoningcited by 1
  26. 2025Mitigating Cross-Image Information Leakage in Multi-Image Understanding with Large Vision-Language Modelscited by 1
  27. 2026MM-THEBench: Do Reasoning MLLMs Think Reasonably?cited by 1
  28. 2026Enhancing Multi-Image Understanding through Delimiter Token Scalingcited by 1
  29. 2026Positional versus Symbolic Attention Heads: Learning Dynamics, RoPE Geometry, and Length Generalizationcited by 1
  30. 2025Adaptive Framework for Ambient Intelligence in Rehabilitation Assistancecited by 0
  31. 2025BlindSight: Harnessing Sparsity for Efficient Vision-Language Modelscited by 0
  32. 2025RealBench: A Chinese Multi-image Understanding Benchmark Close to Real-world Scenarioscited by 0
  33. 2026Mimic Human Cognition, Master Multi-Image Reasoning: A Meta-Action Framework for Enhanced Visual Understandingcited by 0
  34. 2026StemBind: When MLLMs Get Lost Between Rules and Instances in Abstract Visual Reasoningcited by 0
  35. 2026Multi-Image Visual Token Pruning in Large Visual Language Modelscited by 0
  36. 2026CONDUIT: A Unified Residual-Stream Restoration Framework for KV Cache Reuse in Vision-Language Modelscited by 0
  37. 2026SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositoriescited by 0
  38. 2026SWE-PolyVision: Benchmarking Cross-Image Abductive Reasoning for Repository-Level Software Engineeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
  2. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs
  3. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  4. 2025From Easy to Hard: The MIR Benchmark for Progressive Interleaved Multi-Image Reasoning
  5. 2026UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking
  6. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  7. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models
  8. 2025VLURes: Benchmarking VLM Visual and Linguistic Understanding in Low-Resource Languages