mlsanityData to 7 Oct 2026

MOAT: Evaluating LMMs for Capability Integration and Instruction Grounding

arXiv 2503.09348 ↗First posted 12 Mar 2025cs.CL cs.AI cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMMs) have demonstrated significant potential as generalists in vision-language (VL) tasks. However, adoption of LMMs in real-world tasks is hindered by their poor performance in tasks that require a combination of VL capabilities, as well as in tasks that involve the grounding of complex text or visual instructions. To thoroughly investigate this gap and its underlying causes, we propose MOAT, a diverse benchmark with 1005 complex real-world vision questions that are straightforward for humans but challenging for LMMs. Specifically, the tasks in MOAT require LMMs to engage in generalist problem solving by integrating VL capabilities such as reading text, counting, understanding spatial relations, grounding textual and visual instructions, etc. All these abilities fit into a taxonomy proposed by us that contains 9 VL capabilities, enabling MOAT to provide a fine-grained view of LMMs' strengths and weaknesses. Besides, MOAT is the first benchmark to explicitly evaluate LMMs' ability to ground complex text and visual instructions, which is essential for many real-world applications. We evaluated 17 proprietary and open source LMMs, finding that the best performing LMM (Gemini 2.5 Pro) achieved only 44% accuracy, far below what would be acceptable in real-world applications. To guide future model development, we analyze common trends in our results and discuss the underlying causes of poor performance, focusing on the impact of text-centric reasoning, which VL capabilities form bottlenecks in complex tasks, and the potential harmful effects of tiling. Code and data are available at https://cambrian-yzt.github.io/MOAT/.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 87 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

66 of 87 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024NVLM: Open Frontier-Class Multimodal LLMs17 Sep
  5. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?23 Aug
  6. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  7. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  8. 2024MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities1 Aug
  9. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  10. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation29 Jun
  11. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Model31 May
  12. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
7 more
  1. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  3. 2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts1 Dec
  4. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models28 Nov
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  7. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models15 Jun

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2025Qwen2.5-VL Technical Reportcited by 5,418
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2024GPT-4o System Cardcited by 4,467
35 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  11. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  12. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  13. 2025Humanity's Last Examcited by 467
  14. 2024InternLM2 Technical Reportcited by 391
  15. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  16. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
  17. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  18. 2023RSGPT: A Remote Sensing Vision Language Model and Benchmarkcited by 204
  19. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  20. 2025MiniMax-01: Scaling Foundation Models with Lightning Attentioncited by 157
  21. 2025PhysBench: Benchmarking and Enhancing Vision-Language Models for Physical World Understandingcited by 147
  22. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  23. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  24. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  25. 2024NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samplescited by 94
  26. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  27. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  28. 2024WildVision: Evaluating Vision-Language Models in the Wild with Human Preferencescited by 76
  29. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
  30. 2024Large Language Model for Table Processing: A Surveycited by 52
  31. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  32. 2024MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Taskscited by 41
  33. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 38
  34. 2023Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Modelscited by 23
  35. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025InsightX Agent: An LMM-based Agentic Framework with Integrated Tools for Reliable X-ray NDT Analysiscited by 3
  2. 2025Learning to Detect Unknown Jailbreak Attacks in Large Vision-Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  2. 2023On the Performance of Multimodal Language Models
  3. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
  4. 2023Improved Baselines with Visual Instruction Tuning
  5. 2026Mind's Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)
  7. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  8. 2025Understand, Think, and Answer: Advancing Visual Reasoning with Large Multimodal Models