mlsanityData to 14 Oct 2026

MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

arXiv 2406.09411 ↗First posted 13 Jun 2024cs.CV cs.AI cs.CLcited by 188

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the importance of MuirBench in encouraging the community to develop multimodal LLMs that can look beyond a single image, suggesting potential pathways for future improvements.

Key facts. Cited by 188 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 63 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

48 of 63 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  2. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  3. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  4. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  6. 2023Generative Multimodal Models are In-Context Learners20 Dec
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  8. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  9. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning1 Oct
  10. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions8 Aug
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun

Other references · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  10. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
24 more
  1. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  2. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  3. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  4. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  5. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  6. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  7. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
  8. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  9. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  10. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  11. 2021Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoningcited by 383
  12. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  13. 2022Visual Spatial Reasoningcited by 293
  14. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  15. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  16. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  17. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 172
  18. 2020University-1652: A Multi-view Multi-source Benchmark for Drone-based Geo-localizationcited by 157
  19. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  20. 2021D2S: Document-to-Slide Generation Via Query-Based Text Summarizationcited by 50
  21. 2023GeneCIS: A Benchmark for General Conditional Image Similaritycited by 47
  22. 2020Visual Question Answering on Image Setscited by 26
  23. 2022There is a Time and Place for Reasoning Beyond the Imagecited by 22
  24. 2019Deep Learning for Multiple-Image Super-Resolutioncited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 30; 2025: 82; 2026: 76.

188 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  6. 2025Qwen2.5-Omni Technical Reportcited by 736
  7. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  8. 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 466
  9. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  10. 2025MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligencecited by 183
  11. 2025LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuningcited by 148
  12. 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3Dcited by 138
38 more
  1. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  2. 2025MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiencycited by 125
  3. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  4. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
  5. 2024mDPO: Conditional Preference Optimization for Multimodal Large Language Modelscited by 92
  6. 2025VTool-R1: VLMs Learn to Think with Images via Reinforcement Learning on Multimodal Tool Usecited by 90
  7. 2026MiniCPM-o 4.5: Towards Real-Time Full-Duplex Omni-Modal Interactioncited by 78
  8. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  9. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  10. 2025Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokenscited by 71
  11. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  12. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 55
  13. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 50
  14. 2025PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Modelscited by 49
  15. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  16. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  17. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  18. 2025Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Playcited by 37
  19. 2025VisMem: Latent Vision Memory Unlocks Potential of Vision-Language Modelscited by 37
  20. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  21. 2025VLM2-Bench: A Closer Look at How Well VLMs Implicitly Link Explicit Matching Visual Cuescited by 33
  22. 2024Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLMcited by 32
  23. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  24. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 32
  25. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 31
  26. 2025SITE: towards Spatial Intelligence Thorough Evaluationcited by 29
  27. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 28
  28. 2024Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Modelscited by 28
  29. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  30. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 28
  31. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  32. 2025LongCat-Flash-Omni Technical Reportcited by 26
  33. 2024Leveraging Vision Language Models for Specialized Agricultural Taskscited by 25
  34. 2025Identifying and Mitigating Position Bias of Multi-image Vision-Language Modelscited by 25
  35. 2024MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problemscited by 24
  36. 2025Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Modelscited by 22
  37. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 22
  38. 2025Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Modelscited by 21

50 most cited of 188 All 188 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Images
  2. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
  3. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs
  4. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  5. 2024Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion
  6. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Models
  7. 2025IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting
  8. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation