mlsanityData to 14 Oct 2026

Benchmarking Vision Language Models on German Factual Data

arXiv 2504.11108 ↗First posted 15 Apr 2025cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Similar to LLMs, the development of vision language models is mainly driven by English datasets and models trained in English and Chinese language, whereas support for other languages, even those considered high-resource languages such as German, remains significantly weaker. In this work we present an analysis of open-weight VLMs on factual knowledge in the German and English language. We disentangle the image-related aspects from the textual ones by analyzing accu-racy with jury-as-a-judge in both prompt languages and images from German and international contexts. We found that for celebrities and sights, VLMs struggle because they are lacking visual cognition of German image contents. For animals and plants, the tested models can often correctly identify the image contents ac-cording to the scientific name or English common name but fail in German lan-guage. Cars and supermarket products were identified equally well in English and German images across both prompt languages.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 43 references that are in the map. First posted on arXiv 15 Apr 2025.

Builds on

37 of 43 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages25 Nov
  2. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines16 Oct
  3. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models25 Sep
  4. 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
  5. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs20 Jun
  6. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun
  7. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models15 Mar
  8. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  9. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  11. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 26

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2015VQA: Visual Question Answeringcited by 2,506
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  12. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
14 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  2. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  3. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  4. 2023Multimodal Large Language Models: A Surveycited by 195
  5. 2024Pixtral 12Bcited by 179
  6. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 146
  7. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  8. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 97
  9. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  10. 2024MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answeringcited by 93
  11. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  12. 2024Parrot: Multilingual Visual Instruction Tuningcited by 30
  13. 2024M4U: Evaluating Multilingual Understanding and Reasoning for Large Multimodal Modelscited by 13
  14. 2024Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answeringcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025VLM@school -- Evaluation of AI image understanding on German middle school knowledgecited by 2
  2. 2026SITUATE -- Synthetic Object Counting Dataset for VLM trainingcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026V-DyKnow: A Dynamic Benchmark for Time-Sensitive Knowledge in Vision Language Models
  2. 2024Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models
  3. 2025A Multimodal Recaptioning Framework to Account for Perceptual Diversity Across Languages in Vision-Language Modeling
  4. 2026WorldVQA: Measuring Atomic World Knowledge in Multimodal Large Language Models
  5. 2024Building and better understanding vision-language models: insights and future directions
  6. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning
  7. 2025"See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models
  8. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding