mlsanityData to 7 Oct 2026

MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

arXiv 2607.01813 ↗First posted 2 Jul 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Evaluation benchmarks are essential for assessing vision-language models (VLMs), but most multimodal benchmarks are static, making them vulnerable to temporal staleness, data contamination, and costly maintenance. We present MMBench-Live, a continuously evolving multimodal benchmark built by a multi-agent-driven automated pipeline. Our framework treats benchmark evolution as task-guided dataset construction, integrating structured benchmark specification, feedback-controlled real-time data acquisition, and verifiable QA generation with executable reasoning. To maintain cross-version comparability, we introduce a distribution-consistent update strategy that extracts task-related visual patterns from the original benchmark to guide data collection and filtering. Instantiated from MMBench, MMBench-Live contains 5.9K newly generated evaluation instances with a high answer correctness rate, while each update costs about USD 30 and takes 1-2 hours. Extensive evaluations show that MMBench-Live preserves stable model rankings, maintains semantic alignment with the original benchmark, and exhibits weaker contamination-related memorization signals, suggesting a practical and scalable paradigm for sustainable multimodal benchmark evolution. The project is available at https://github.com/PRIS-CV/MMBench-Live.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 36 references that are in the map. First posted on arXiv 2 Jul 2026.

Builds on

30 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding14 Aug
  2. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  5. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 25

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2024Depth Anything V2cited by 1,385
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  9. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  10. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  11. 2024General OCR Theory: Towards OCR-2.0 via a Unified End-to-end Modelcited by 174
  12. 2023On the Tool Manipulation Capability of Open-source Large Language Modelscited by 122
13 more
  1. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  2. 2025Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanationcited by 66
  3. 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learningcited by 50
  4. 2024AntiLeakBench: Preventing Data Contamination by Automatically Constructing Benchmarks with Updated Real-World Knowledgecited by 42
  5. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrappingcited by 22
  6. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  7. 2025Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination's Impact on Machine Translationcited by 13
  8. 2024Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMcited by 9
  9. 2024BenchmarkCards: Standardized Documentation for Large Language Model Benchmarkscited by 8
  10. 2024QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarizationcited by 8
  11. 2025Acknowledging Focus Ambiguity in Visual Questionscited by 7
  12. 2024Dysca: A Dynamic and Scalable Benchmark for Evaluating Perception Ability of LVLMscited by 6
  13. 2024API Pack: A Massive Multi-Programming Language Dataset for API Call Generationcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  3. 2026VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining
  4. 2026WorldBench: A Challenging and Visually Diverse Multimodal Reasoning Benchmark
  5. 2024MileBench: Benchmarking MLLMs in Long Context
  6. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  7. 2026MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledge
  8. 2026AdaptMMBench: Benchmarking Adaptive Multimodal Reasoning for Mode Selection and Reasoning Process