mlsanityData to 7 Oct 2026

How Culturally Aware are Vision-Language Models?

arXiv 2405.17475 ↗First posted 24 May 2024cs.CV cs.AI cs.CLcited by 6

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

An image is often considered worth a thousand words, and certain images can tell rich and insightful stories. Can these stories be told via image captioning? Images from folklore genres, such as mythology, folk dance, cultural signs, and symbols, are vital to every culture. Our research compares the performance of four popular vision-language models (GPT-4V, Gemini Pro Vision, LLaVA, and OpenFlamingo) in identifying culturally specific information in such images and creating accurate and culturally sensitive image captions. We also propose a new evaluation metric, the Cultural Awareness Score (CAS), which measures the degree of cultural awareness in image captions. We provide a dataset MOSAIC-1.5k labeled with ground truth for images containing cultural background and context and a labeled dataset with assigned Cultural Awareness Scores that can be used with unseen data. Creating culturally appropriate image captions is valuable for scientific research and can be beneficial for many practical applications. We envision our work will promote a deeper integration of cultural sensitivity in AI applications worldwide. By making the dataset and Cultural Awareness Score available to the public, we aim to facilitate further research in this area, encouraging the development of more culturally aware AI systems that respect and celebrate global diversity.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 18 of its 39 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

18 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing8 Feb
  2. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models2 Aug
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 15

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  6. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  7. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  8. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  9. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  10. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  11. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  12. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Modelscited by 223
3 more
  1. 2023TouchStone: Evaluating Vision-Language Models by Language Modelscited by 54
  2. 2021Iconographic Image Captioning for Artworkscited by 15
  3. 2023IMAGINATOR: Pre-Trained Image+Text Joint Embeddings using Word-Level Grounding of Imagescited by 4

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 3.

6 papers.

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2024Vision-Language Models under Cultural and Inclusive Considerationscited by 8
  3. 2025Pearl: A Multimodal Culturally-Aware Arabic Instruction Datasetcited by 6
  4. 2024KAHANI: Culturally-Nuanced Visual Storytelling Tool for Non-Western Culturescited by 4
  5. 2025World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Modelscited by 3
  6. 2025Culture In a Frame: C3B as a Comic-Based Benchmark for Multimodal Culturally Awarenesscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  2. 2024Benchmarking Vision Language Models for Cultural Understanding
  3. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
  4. 2025Evaluation of Cultural Competence of Vision-Language Models
  5. 2025Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation
  6. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding
  7. 2025EgMM-Corpus: A Multimodal Vision-Language Dataset for Egyptian Culture
  8. 2024CIC: A Framework for Culturally-Aware Image Captioning