# Redundancy Principles for MLLMs Benchmarks

arXiv: https://arxiv.org/abs/2501.13953 · HTML: https://mlsanity.com/paper/2501.13953/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 20 Jan 2025.

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset](https://mlsanity.com/paper/2402.14804/) (2024)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision](https://mlsanity.com/paper/2309.14181/) (2023)
- [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://mlsanity.com/paper/2305.18752/) (2023)
- [MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI](https://mlsanity.com/paper/2404.16006/) (2024)
- [OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM](https://mlsanity.com/paper/2402.09181/) (2024)
- [SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension](https://mlsanity.com/paper/2404.16790/) (2024)
- [VRSBench: A Versatile Vision-Language Benchmark Dataset for Remote Sensing Image Understanding](https://mlsanity.com/paper/2406.12384/) (2024)
- [ScreenAI: A Vision-Language Model for UI and Infographics Understanding](https://mlsanity.com/paper/2402.04615/) (2024)
- [A-Bench: Are LMMs Masters at Evaluating AI-generated Images?](https://mlsanity.com/paper/2406.03070/) (2024)
- [R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions?](https://mlsanity.com/paper/2410.05474/) (2024)

## Built on by (most cited first, 19 of 19)

- [LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?](https://mlsanity.com/paper/2503.19990/) (2025)
- [Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM](https://mlsanity.com/paper/2503.14478/) (2025)
- [Information Density Principle for MLLM Benchmarks](https://mlsanity.com/paper/2503.10079/) (2025)
- [Jigsaw-Puzzles: From Seeing to Understanding to Reasoning in Vision-Language Models](https://mlsanity.com/paper/2505.20728/) (2025)
- [VER-Bench: Evaluating MLLMs on Reasoning with Fine-Grained Visual Evidence](https://mlsanity.com/paper/2508.04852/) (2025)
- [Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data?](https://mlsanity.com/paper/2510.10457/) (2025)
- [A2Eval: Agentic and Automated Evaluation for Embodied Brain](https://mlsanity.com/paper/2602.01640/) (2026)
- [Improve MLLM Benchmark Efficiency through Interview](https://mlsanity.com/paper/2506.00883/) (2025)
- [The Ever-Evolving Science Exam](https://mlsanity.com/paper/2507.16514/) (2025)
- [STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction](https://mlsanity.com/paper/2602.12143/) (2026)
- [BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity](https://mlsanity.com/paper/2603.18019/) (2026)
- [Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation](https://mlsanity.com/paper/2609.04298/) (2026)
- [Aligning MLLM Benchmark With Human Preferences via Structural Equation Modeling](https://mlsanity.com/paper/2506.21572/) (2025)
- [A Multi-To-One Interview Paradigm for Efficient MLLM Evaluation](https://mlsanity.com/paper/2509.14886/) (2025)
- [UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities](https://mlsanity.com/paper/2603.23160/) (2026)
- [An Interpretable and Scalable Framework for Evaluating Large Language Models](https://mlsanity.com/paper/2605.07046/) (2026)
- [CoT-Core: Accelerating LLM Evaluation via CoT-Aware Coreset Selection](https://mlsanity.com/paper/2608.00014/) (2026)
- [Modality Maturity Index: A benchmark for assessing multimodal capabilities of omni models](https://mlsanity.com/paper/2608.26317/) (2026)
- [Auditing VLM Benchmarks from Existing Evaluation Runs](https://mlsanity.com/paper/2610.13548/) (2026)

## Abstract (arXiv)

With the rapid iteration of Multi-modality Large Language Models (MLLMs) and the evolving demands of the field, the number of benchmarks produced annually has surged into the hundreds. The rapid growth has inevitably led to significant redundancy among benchmarks. Therefore, it is crucial to take a step back and critically assess the current state of redundancy and propose targeted principles for constructing effective MLLM benchmarks. In this paper, we focus on redundancy from three key perspectives: 1) Redundancy of benchmark capability dimensions, 2) Redundancy in the number of test questions, and 3) Cross-benchmark redundancy within specific domains. Through the comprehensive analysis over hundreds of MLLMs' performance across more than 20 benchmarks, we aim to quantitatively measure the level of redundancy lies in existing MLLM evaluations, provide valuable insights to guide the future development of MLLM benchmarks, and offer strategies to refine and address redundancy issues effectively. The code is available at https://github.com/zzc-1998/Benchmark-Redundancy.