mlsanityData to 7 Oct 2026

What We are Missing in Multimodal LLM Evaluation?

arXiv 2606.26348 ↗First posted 24 Jun 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) can process diverse inputs, e.g., text, images, audio, and video, and generate textual responses. While their capabilities have advanced rapidly, evaluation of such models has not kept pace. Most existing evaluation benchmarks are limited to isolated tasks and reveal little about whether a model integrates information across modalities. We examine current means for evaluating MLLMs and review the existing benchmark taxonomy to identify gaps, including temporal-spatial coherence, physical world understanding, multimodal consistency, and selective attention. Addressing these gaps is essential for measuring real progress in multimodal intelligence and exposing capability boundaries.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 7 of its 10 references that are in the map. First posted on arXiv 24 Jun 2026.

Builds on

7 of 10 references.

References · 7

  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  2. 2017Dense-Captioning Events in Videoscited by 789
  3. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  4. 2025Why Language Models Hallucinatecited by 205
  5. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  6. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  7. 2025VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DRcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs
  2. 2024Understanding the Role of LLMs in Multimodal Evaluation Benchmarks
  3. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
  5. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  6. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  7. 2024A Survey on Benchmarks of Multimodal Large Language Models
  8. 2023MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks