# Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping

arXiv: https://arxiv.org/abs/2410.08695 · HTML: https://mlsanity.com/paper/2410.08695/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 37 of its 43 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- Synthetic data and self-improvement for LLMs / [Self-improving MLLMs and synthetic data](https://mlsanity.com/synth/self-improving-mllms-and-synthetic-data/)

## Builds on (most cited first, 30 of 37)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs](https://mlsanity.com/paper/2111.02114/) (2021)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge](https://mlsanity.com/paper/1906.00067/) (2019)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models](https://mlsanity.com/paper/2311.06607/) (2023)
- [InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model](https://mlsanity.com/paper/2401.16420/) (2024)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Models](https://mlsanity.com/paper/2310.00746/) (2023)
- [A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting](https://mlsanity.com/paper/2312.03594/) (2023)
- [SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models](https://mlsanity.com/paper/2402.05935/) (2024)

## Built on by (most cited first, 22 of 22)

- [Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs](https://mlsanity.com/paper/2504.15280/) (2025)
- [MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2504.05782/) (2025)
- [TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models](https://mlsanity.com/paper/2506.01341/) (2025)
- [Truly Assessing Fluid Intelligence of Large Language Models through Dynamic Reasoning Evaluation](https://mlsanity.com/paper/2506.02648/) (2025)
- [Can Large Models Fool the Eye? A New Turing Test for Biological Animation](https://mlsanity.com/paper/2508.06072/) (2025)
- [TestAgent: Automatic Benchmarking and Exploratory Interaction for Evaluating LLMs in Vertical Domains](https://mlsanity.com/paper/2410.11507/) (2024)
- [MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams](https://mlsanity.com/paper/2508.06851/) (2025)
- [SDEval: Safety Dynamic Evaluation for Multimodal Large Language Models](https://mlsanity.com/paper/2508.06142/) (2025)
- [Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory](https://mlsanity.com/paper/2603.02663/) (2026)
- [KBE-DME: Dynamic Multimodal Evaluation via Knowledge Enhanced Benchmark Evolution](https://mlsanity.com/paper/2510.21182/) (2025)
- [MACEval: A Multi-Agent Continual Evaluation Network for Large Models](https://mlsanity.com/paper/2511.09139/) (2025)
- [Evaluating from Benign to Dynamic Adversarial: A Squid Game for Large Language Models](https://mlsanity.com/paper/2511.10691/) (2025)
- [Automatic Cognitive Task Generation for In-Situ Evaluation of Embodied Agents](https://mlsanity.com/paper/2602.05249/) (2026)
- [LEAF: A Living Benchmark for Event-Augmented Forecasting](https://mlsanity.com/paper/2605.16358/) (2026)
- [SafetyFlow: An Agent-Flow System for Automated LLM Safety Benchmarking](https://mlsanity.com/paper/2508.15526/) (2025)
- [Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMs](https://mlsanity.com/paper/2601.19507/) (2026)
- [Dynamic Adversarial Reinforcement Learning for Robust Multimodal Large Language Models](https://mlsanity.com/paper/2602.22227/) (2026)
- [DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoning](https://mlsanity.com/paper/2605.29339/) (2026)
- [REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluation](https://mlsanity.com/paper/2606.20736/) (2026)
- [MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models](https://mlsanity.com/paper/2607.01813/) (2026)
- [SABRE: Scalable and Automated Benchmarking of VLMs under Stress](https://mlsanity.com/paper/2608.07435/) (2026)
- [Improving Generalization Robustness of Multimodal RLVR](https://mlsanity.com/paper/2608.08802/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities across multimodal tasks such as visual perception and reasoning, leading to good performance on various multimodal evaluation benchmarks. However, these benchmarks keep a static nature and overlap with the pre-training data, resulting in fixed complexity constraints and data contamination issues. This raises the concern regarding the validity of the evaluation. To address these two challenges, we introduce a dynamic multimodal evaluation protocol called Vision-Language Bootstrapping (VLB). VLB provides a robust and comprehensive assessment for LVLMs with reduced data contamination and flexible complexity. To this end, VLB dynamically generates new visual question-answering samples through a multimodal bootstrapping module that modifies both images and language, while ensuring that newly generated samples remain consistent with the original ones by a judge module. By composing various bootstrapping strategies, VLB offers dynamic variants of existing benchmarks with diverse complexities, enabling the evaluation to co-evolve with the ever-evolving capabilities of LVLMs. Extensive experimental results across multiple benchmarks, including SEEDBench, MMBench, and MME, show that VLB significantly reduces data contamination and exposes performance limitations of LVLMs.