mlsanityData to 7 Oct 2026

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

arXiv 2408.02718 ↗First posted 5 Aug 2024cs.CVcited by 61

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The capability to process multiple images is crucial for Large Vision-Language Models (LVLMs) to develop a more thorough and nuanced understanding of a scene. Recent multi-image LVLMs have begun to address this need. However, their evaluation has not kept pace with their development. To fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess LVLMs across a wide range of multi-image tasks. MMIU encompasses 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions, making it the most extensive benchmark of its kind. Our evaluation of 24 popular LVLMs, including both open-source and proprietary models, reveals significant challenges in multi-image comprehension, particularly in tasks involving spatial understanding. Even the most advanced models, such as GPT-4o, achieve only 55.7% accuracy on MMIU. Through multi-faceted analytical experiments, we identify key performance gaps and limitations, providing valuable insights for future model and data improvements. We aim for MMIU to advance the frontier of LVLM research and development, moving us toward achieving sophisticated multimodal multi-image user interactions.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 101 references that are in the map. First posted on arXiv 5 Aug 2024.

Builds on

75 of 101 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  2. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding13 Jun
  3. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  4. 2024MileBench: Benchmarking MLLMs in Long Context29 Apr
  5. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  6. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  7. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  8. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model29 Jan
  9. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  10. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  11. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models11 Nov
  12. 2023Improved Baselines with Visual Instruction Tuning5 Oct
3 more
  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  2. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
  3. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun

Other references · 60

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  7. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2017The Kinetics Human Action Video Datasetcited by 1,894
  10. 2024Mixtral of Expertscited by 1,785
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
38 more
  1. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  2. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
  3. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  4. 2019FaceForensics++: Learning to Detect Manipulated Facial Imagescited by 1,095
  5. 2022Editing Models with Task Arithmeticcited by 1,076
  6. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  7. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  8. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  9. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  11. 2016NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysiscited by 683
  12. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  13. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  14. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  15. 2019Video Instance Segmentationcited by 391
  16. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  17. 2024What matters when building vision-language models?cited by 321
  18. 2017A Neural Representation of Sketch Drawingscited by 311
  19. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  20. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  21. 2017HPatches: A benchmark and evaluation of handcrafted and learned local descriptorscited by 261
  22. 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 258
  23. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  24. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 247
  25. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  26. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  27. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  28. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  29. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  30. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  31. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  32. 2018Measuring abstract reasoning in neural networkscited by 165
  33. 2021SPEC: Seeing People in the Wild with an Estimated Cameracited by 134
  34. 2021ForgeryNet: A Versatile Benchmark for Comprehensive Forgery Analysiscited by 124
  35. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  36. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
  37. 2017PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understandingcited by 101
  38. 2016Learning To Score Olympic Eventscited by 87

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 36; 2026: 18.

61 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  4. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 169
  5. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  6. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  7. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  8. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 31
  9. 2024Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignmentcited by 29
  10. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  11. 2025SITE: towards Spatial Intelligence Thorough Evaluationcited by 26
  12. 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Modelscited by 22
38 more
  1. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmarkcited by 20
  2. 2024AutoBench-V: Can Large Vision-Language Models Benchmark Themselves?cited by 19
  3. 2025SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligencecited by 19
  4. 2025Multi-Agent System for Comprehensive Soccer Understandingcited by 18
  5. 2025DynamicVL: Benchmarking Multimodal Large Language Models for Dynamic City Understandingcited by 17
  6. 2025Are They the Same? Exploring Visual Correspondence Shortcomings of Multimodal LLMscited by 16
  7. 2025Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Modelscited by 13
  8. 2025RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Eventscited by 13
  9. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  10. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understandingcited by 12
  11. 2025PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoningcited by 11
  12. 2025Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learningcited by 11
  13. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 11
  14. 2025Medical Large Vision Language Models with Multi-Image Visual Abilitycited by 8
  15. 2025VisioMath: Benchmarking Figure-based Mathematical Reasoning in LMMscited by 6
  16. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  17. 2025A Cognitive Paradigm Approach to Probe the Perception-Reasoning Interface in VLMscited by 5
  18. 2025ConViS-Bench: Estimating Video Similarity Through Semantic Conceptscited by 4
  19. 2025SpatialMosaic: A Multiview VLM Dataset for Partial Visibilitycited by 4
  20. 2025SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoningcited by 3
  21. 2025Revisiting Multimodal KV Cache Compression: A Frequency-Domain-Guided Outlier-KV-Aware Approachcited by 3
  22. 2025VisChainBench: A Benchmark for Multi-Turn, Multi-Image Visual Reasoning Beyond Language Priorscited by 3
  23. 2026MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documentscited by 3
  24. 2026OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Modelcited by 3
  25. 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 3
  26. 2025Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMscited by 2
  27. 2025SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Servicescited by 2
  28. 2025See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosiscited by 2
  29. 2025HumanPCR: Probing MLLM Capabilities in Diverse Human-Centric Scenescited by 2
  30. 2026ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Modelscited by 2
  31. 2026CRIT: Graph-Based Automatic Data Synthesis to Enhance Cross-Modal Multi-Hop Reasoningcited by 2
  32. 2026IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Productscited by 2
  33. 2025Human-Aligned Bench: Fine-Grained Assessment of Reasoning Ability in MLLMs vs. Humanscited by 1
  34. 2025MANBench: Is Your Multimodal Model Smarter than Human?cited by 1
  35. 2025QG-CoC: Question-Guided Chain-of-Captions for Large Multimodal Modelscited by 1
  36. 2025AVFakeBench: A Comprehensive Audio-Video Forgery Detection Benchmark for AV-LMMscited by 1
  37. 2026Contextualized Visual Personalization in Vision-Language Modelscited by 1
  38. 2026Unified Spatio-Temporal Token Scoring for Efficient Video VLMscited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  2. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models
  3. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
  4. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks
  5. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  6. 2025MMLongCite: A Benchmark for Evaluating Fidelity of Long-Context Vision-Language Models
  7. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities
  8. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly