mlsanityData to 7 Oct 2026

MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMs

arXiv 2407.16837 ↗First posted 23 Jul 2024cs.CV cs.AI cs.CLcited by 32

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

The ability to compare objects, scenes, or situations is crucial for effective decision-making and problem-solving in everyday life. For instance, comparing the freshness of apples enables better choices during grocery shopping while comparing sofa designs helps optimize the aesthetics of our living space. Despite its significance, the comparative capability is largely unexplored in artificial general intelligence (AGI). In this paper, we introduce MLLM-CompBench, a benchmark designed to evaluate the comparative reasoning capability of multimodal large language models (MLLMs). MLLM-CompBench mines and pairs images through visually oriented questions covering eight dimensions of relative comparison: visual attribute, existence, state, emotion, temporality, spatiality, quantity, and quality. We curate a collection of around 40K image pairs using metadata from diverse vision datasets and CLIP similarity scores. These image pairs span a broad array of visual domains, including animals, fashion, sports, and both outdoor and indoor scenes. The questions are carefully crafted to discern relative characteristics between two images and are labeled by human annotators for accuracy and relevance. We use MLLM-CompBench to evaluate recent MLLMs, including GPT-4V(ision), Gemini-Pro, and LLaVA-1.6. Our results reveal notable shortcomings in their comparative abilities. We believe MLLM-COMPBENCH not only sheds light on these limitations but also establishes a solid foundation for future enhancements in the comparative capability of MLLMs.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 12 of its 70 references that are in the map. First posted on arXiv 23 Jul 2024.

Builds on

12 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ReMI: A Dataset for Reasoning with Multiple Images13 Jun

Other references · 11

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  8. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 212
  11. 2024Q-Bench+: A Benchmark for Multi-modal Foundation Models on Low-level Vision from Single Images to Pairscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 18; 2026: 10.

32 papers.

Sort:

By area:

  1. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  2. 2025Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Modelscited by 43
  3. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 28
  4. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  5. 2026UniCorn: Towards Self-Improving Unified Multimodal Models through Self-Generated Supervisioncited by 23
  6. 2025Benchmarking Multimodal Knowledge Conflict for Large Multimodal Modelscited by 18
  7. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 11
  8. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  9. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?cited by 6
  10. 2025AVA-Bench: Atomic Visual Ability Benchmark for Vision Foundation Modelscited by 5
  11. 2025Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Studycited by 4
  12. 2025VOILA: Evaluation of MLLMs For Perceptual Understanding and Analogical Reasoningcited by 4
20 more
  1. 2025MV-Debate: Multi-view Agent Debate with Dynamic Reflection Gating for Multimodal Harmful Content Detection in Social Mediacited by 4
  2. 2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Modelscited by 3
  3. 2025MedBookVQA: A Systematic and Comprehensive Medical Benchmark Derived from Open-Access Bookcited by 2
  4. 2025Breaking Bad Molecules: Are MLLMs Ready for Structure-Level Molecular Detoxification?cited by 2
  5. 2025MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Modelscited by 2
  6. 2025Multimodal LLMs See Sentimentcited by 2
  7. 2025Towards Scalable Web Accessibility Audit with MLLMs as Copilotscited by 2
  8. 2024Project MPG: towards a generalized performance benchmark for LLM capabilitiescited by 1
  9. 2025A Lightweight Framework for Trigger-Guided LoRA-Based Self-Adaptation in LLMscited by 1
  10. 2026MIDAS: Multi-Image Dispersion and Semantic Reconstruction for Jailbreaking MLLMscited by 1
  11. 2026Granulon: Awakening Pixel-Level Visual Encoders with Adaptive Multi-Granularity Semantics for MLLMcited by 1
  12. 2024Role-RL: Online Long-Context Processing with Role Reinforcement Learning for Distinct LLMs in Their Optimal Rolescited by 0
  13. 2025Retrieval Enhanced Feedback via In-context Neural Error-bookcited by 0
  14. 2026Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Taskcited by 0
  15. 2026AMIGO: Agentic Multi-Image Grounding Oracle Benchmarkcited by 0
  16. 2026AgroTools: A Benchmark for Tool-Augmented Multimodal Agents in Agriculturecited by 0
  17. 2026Does VLA Even Know the Basics? Measuring Commonsense and World Knowledge Retention in Vision-Language-Action Modelscited by 0
  18. 2026DiCoBench: Benchmarking Multi-Image Fine-Grained Perception via Differential and Commonality Visual Cuescited by 0
  19. 2026POPS: Recovering Unlearned Multi-Modality Knowledge in MLLMs with Prompt-Optimized Parameter Shakingcited by 0
  20. 2026Activation Outliers Matter: Robust Recovery for Quantized Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
  2. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models
  3. 2026WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
  4. 2025CompareBench: A Benchmark for Visual Comparison Reasoning in Vision-Language Models
  5. 2025All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark
  6. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
  7. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks
  8. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models