mlsanityData to 14 Oct 2026

Evaluation of Cultural Competence of Vision-Language Models

arXiv 2505.22793 ↗First posted 28 May 2025cs.CV cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Modern vision-language models (VLMs) often fail at cultural competency evaluations and benchmarks. Given the diversity of applications built upon VLMs, there is renewed interest in understanding how they encode cultural nuances. While individual aspects of this problem have been studied, we still lack a comprehensive framework for systematically identifying and annotating the nuanced cultural dimensions present in images for VLMs. This position paper argues that foundational methodologies from visual culture studies (cultural studies, semiotics, and visual studies) are necessary for cultural analysis of images. Building upon this review, we propose a set of five frameworks, corresponding to cultural dimensions, that must be considered for a more complete analysis of the cultural competencies of VLMs.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 104 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

34 of 104 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding20 May
  2. 2025GIMMICK -- Globally Inclusive Multimodal Multitask Cultural Knowledge Benchmarking19 Feb
  3. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries2 Jan
  4. 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
  5. 2024Vision-Language Models under Cultural and Inclusive Considerations8 Jul
  6. 2024Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration24 Jun
  7. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun
  8. 2024Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing8 Feb
  9. 2024CIC: A Framework for Culturally-Aware Image Captioning8 Feb

Other references · 25

  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  2. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  3. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  4. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  5. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  6. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  7. 2023SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Modelscited by 66
  8. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  9. 2023GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognitioncited by 50
  10. 2024Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them incited by 46
  11. 2025Culture is Not Trivia: Sociocultural Theory for Cultural NLPcited by 40
  12. 2024An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevancecited by 36
13 more
  1. 2023AltDiffusion: A Multilingual Text-to-Image Diffusion Modelcited by 30
  2. 2024FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culturecited by 25
  3. 2024Extrinsic Evaluation of Cultural Competence in Large Language Modelscited by 24
  4. 2024Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learningcited by 19
  5. 2024No Culture Left Behind: ArtELingo-28, a Benchmark of WikiArt with Captions in 28 Languagescited by 19
  6. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Modelscited by 14
  7. 2023Towards Equitable Representation in Text-to-Image Synthesis Models with the Cross-Cultural Understanding Benchmark (CCUB) Datasetcited by 12
  8. 2023HaVQA: A Dataset for Visual Question Answering and Multimodal Research in Hausa Languagecited by 12
  9. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  10. 2024ViTextVQA: A Large-Scale Visual Question Answering Dataset and a Novel Multimodal Feature Fusion Method for Vietnamese Text Comprehension in Imagescited by 7
  11. 2024Navigating Text-to-Image Generative Bias across Indic Languagescited by 5
  12. 2024CAP: Evaluation of Persuasive and Creative Image Generationcited by 5
  13. 2025Deconstructing Bias: A Multifaceted Framework for Diagnosing Cultural and Compositional Inequities in Text-to-Image Generative Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026A Unified Framework to Quantify Cultural Intelligence of AIcited by 2
  2. 2026Calibrated Ambiguity in Multimodal Language Models: Humans reach for cultural references, while models describe the picturecited by 0
  3. 2026From annotation to reasoning: Culture in language modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
  2. 2026VULCA-Bench: A Multicultural Vision-Language Benchmark for Evaluating Cultural Understanding
  3. 2025Vision Language Models are Confused Tourists
  4. 2025Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation
  5. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding
  6. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
  7. 2025Uncovering Cultural Representation Disparities in Vision-Language Models
  8. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models