mlsanityData to 14 Oct 2026

Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling

arXiv 2506.21572 ↗First posted 13 Jun 2025cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Evaluating multimodal large language models (MLLMs) is fundamentally challenged by the absence of structured, interpretable, and theoretically grounded benchmarks; current heuristically-grouped tasks have vague cognitive targets, overlapping abilities, redundant indicators, and weak diagnostic power. We therefore propose a structural-equation-modeling-aligned framework that quantifies internal validity, dimensional separability, and component contributions, and introduce a Piaget-inspired capability hierarchy that stratifies MLLM abilities into Perception, Memory, and Reasoning. Reorganizing existing tasks under this theory, we build the GOLD benchmark, whose experiments show superior interpretability, lower indicator redundancy, and clearer cognitive consistency than prior benchmarks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 51 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

38 of 51 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025Information Density Principle for MLLM Benchmarks13 Mar
  2. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models13 Feb
  3. 2025Benchmarking Large and Small MLLMs4 Jan
  4. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs22 Nov
  5. 2024OmniBench: Towards The Future of Universal Omni-Language Models23 Sep
  6. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models21 Sep
  7. 2024LIME: Less Is More for MLLM Evaluation10 Sep
  8. 2024A Survey on Evaluation of Multimodal Large Language Models28 Aug
  9. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  10. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 26

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  10. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  11. 2025Gemma 3 Technical Reportcited by 1,824
  12. 2019Towards VQA Models That Can Readcited by 1,718
14 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  5. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  7. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  8. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  9. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 144
  10. 2022A Benchmark for Compositional Visual Reasoningcited by 43
  11. 2025R-Bench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluationcited by 34
  12. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scalingcited by 31
  13. 2025Redundancy Principles for MLLMs Benchmarkscited by 19
  14. 2025EnigmaEval: A Benchmark of Long Multimodal Reasoning Challengescited by 19

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluating and Advancing Multimodal Large Language Models in Perception Ability Lens
  2. 2025On Path to Multimodal Generalist: General-Level and General-Bench
  3. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
  4. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  5. 2024Assessment of Multimodal Large Language Models in Alignment with Human Values
  6. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Model
  7. 2026MLLM-as-a-Judge Exhibits Model Preference Bias
  8. 2023Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Models