mlsanityData to 7 Oct 2026

KBE-DME: Dynamic Multimodal Evaluation via Knowledge Enhanced Benchmark Evolution

arXiv 2510.21182 ↗First posted 24 Oct 2025cs.CV cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The rapid progress of multimodal large language models (MLLMs) calls for more reliable evaluation protocols. Existing static benchmarks suffer from the potential risk of data contamination and saturation, leading to inflated or misleading performance evaluations. To address these issues, we first apply Graph formulation to represent a static or dynamic VQA sample. With the formulation, we propose Knowledge-enhanced Benchmark Evolution(KBE), a dynamic multimodal evaluation framework. KBE first analyzes the original static benchmark, then expands it by integrating multimodal knowledge, transforming the static benchmark into a controllable, dynamic evolving version. Crucially, KBE can both reconstruct questions by Re-selecting visual information in the original image and expand existing questions with external textual knowledge. It enables difficulty-controllable evaluation by adjusting the degree of question exploration. Extensive experiments demonstrate that KBE alleviates the risk of data contamination, data saturation, and provides a more comprehensive assessment of MLLM capabilities.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 20 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

19 of 20 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation8 Jun
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 15

  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2019Towards VQA Models That Can Readcited by 1,696
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  4. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  5. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  8. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  9. 2019Cycle-Consistency for Robust Visual Question Answeringcited by 105
  10. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  11. 2023NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classescited by 78
  12. 2020VQA-LOL: Visual Question Answering under the Lens of Logiccited by 48
3 more
  1. 2024Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrappingcited by 22
  2. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  3. 2024Both Text and Images Leaked! A Systematic Analysis of Data Contamination in Multimodal LLMcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026DMC-CF: Dynamic Multimodal CounterFactual QA benchmark for Causal Reasoningcited by 0
  2. 2026REKEY: Metadata-Grounded Visual-Key Regeneration for Contamination-Resilient VQA Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Understanding the Role of LLMs in Multimodal Evaluation Benchmarks
  2. 2025MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams
  3. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
  4. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  5. 2025Evaluating Variance in Visual Question Answering Benchmarks
  6. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  8. 2024A Survey on Benchmarks of Multimodal Large Language Models