mlsanityData to 14 Oct 2026

Information Density Principle for MLLM Benchmarks

arXiv 2503.10079 ↗First posted 13 Mar 2025cs.CLcited by 9

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

With the emergence of Multimodal Large Language Models (MLLMs), hundreds of benchmarks have been developed to ensure the reliability of MLLMs in downstream tasks. However, the evaluation mechanism itself may not be reliable. For developers of MLLMs, questions remain about which benchmark to use and whether the test results meet their requirements. Therefore, we propose a critical principle of Information Density, which examines how much insight a benchmark can provide for the development of MLLMs. We characterize it from four key dimensions: (1) Fallacy, (2) Difficulty, (3) Redundancy, (4) Diversity. Through a comprehensive analysis of more than 10,000 samples, we measured the information density of 19 MLLM benchmarks. Experiments show that using the latest benchmarks in testing can provide more insight compared to previous ones, but there is still room for improvement in their information density. We hope this principle can promote the development and application of future MLLM benchmarks. Project page: https://github.com/lcysyzxdxc/bench4bench

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 50 references that are in the map. First posted on arXiv 13 Mar 2025.

Builds on

41 of 50 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models13 Feb
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?7 Oct
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  5. 2024LIME: Less Is More for MLLM Evaluation10 Sep
  6. 2024Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models28 Aug
  7. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?23 Aug
  8. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  9. 2024Task Me Anything17 Jun
  10. 2024Parrot: Multilingual Visual Instruction Tuning4 Jun
  11. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension25 Apr
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
3 more
  1. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023Segment Anythingcited by 8,234
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  10. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  12. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
14 more
  1. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  2. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  3. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  4. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  5. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  6. 2023AGIQA-3K: An Open Database for AI-Generated Image Quality Assessmentcited by 142
  7. 2024StableToolBench: Towards Stable Large-Scale Benchmarking on Tool Learning of Large Language Modelscited by 132
  8. 2024A-Bench: Are LMMs Masters at Evaluating AI-generated Images?cited by 39
  9. 2024MISC: Ultra-low Bitrate Image Semantic Compression Driven by Large Multimodal Modelcited by 30
  10. 2025Redundancy Principles for MLLMs Benchmarkscited by 19
  11. 2025EnigmaEval: A Benchmark of Long Multimodal Reasoning Challengescited by 19
  12. 2022A Coding Framework and Benchmark towards Low-Bitrate Video Understandingcited by 17
  13. 2025Towards All-in-One Medical Image Re-Identificationcited by 6
  14. 2024Medical Manifestation-Aware De-Identificationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?cited by 8
  2. 2025Using GUI Agent for Electronic Design Automationcited by 5
  3. 2025Semantics versus Identity: A Divide-and-Conquer Approach towards Adjustable Medical Image De-Identificationcited by 2
  4. 2026Benchmarking PhD-Level Coding in 3D Geometric Computer Visioncited by 2
  5. 2025ROFI: A Deep Learning-Based Ophthalmic Sign-Preserving and Reversible Patient Face Anonymizercited by 1
  6. 2025Aligning MLLM Benchmark With Human Preferences via Structural Equation Modelingcited by 0
  7. 2025QoNext: Towards Next-generation QoE for Foundation Modelscited by 0
  8. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  9. 2026Auditing VLM Benchmarks from Existing Evaluation Runscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding Multimodal LLMs Under Distribution Shifts: An Information-Theoretic Approach
  2. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  3. 2025Improve MLLM Benchmark Efficiency through Interview
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
  5. 2023MLLM-DataEngine: An Iterative Refinement Approach for MLLM
  6. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  7. 2024A Survey on Benchmarks of Multimodal Large Language Models
  8. 2026MLLM-as-a-Judge Exhibits Model Preference Bias