mlsanityData to 7 Oct 2026

Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing

arXiv 2402.06015 ↗First posted 8 Feb 2024cs.CL cs.CVcited by 12

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Pretrained large Vision-Language models have drawn considerable interest in recent years due to their remarkable performance. Despite considerable efforts to assess these models from diverse perspectives, the extent of visual cultural awareness in the state-of-the-art GPT-4V model remains unexplored. To tackle this gap, we extensively probed GPT-4V using the MaRVL benchmark dataset, aiming to investigate its capabilities and limitations in visual understanding with a focus on cultural aspects. Specifically, we introduced three visual related tasks, i.e. caption classification, pairwise captioning, and culture tag selection, to systematically delve into fine-grained visual cultural evaluation. Experimental results indicate that GPT-4V excels at identifying cultural concepts but still exhibits weaker performance in low-resource languages, such as Tamil and Swahili. Notably, through human evaluation, GPT-4V proves to be more culturally relevant in image captioning tasks than the original MaRVL human annotations, suggesting a promising solution for future visual cultural benchmark construction.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 31 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

17 of 31 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs13 Jul

Other references · 16

  1. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  2. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  3. 2020Artificial Intelligence, Values and Alignmentcited by 443
  4. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
  5. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  6. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  7. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  8. 2023Improving CLIP Training with Language Rewritescited by 216
  9. 2023Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Studycited by 203
  10. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  11. 2022Probing Pre-Trained Language Models for Cross-Cultural Differences in Valuescited by 147
  12. 2022The Ghost in the Machine has an American accent: value conflict in GPT-3cited by 105
4 more
  1. 2023VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generationcited by 41
  2. 2023Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learningcited by 16
  3. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  4. 2018Modelling the Influence of Cultural Information on Vision-Based Human Home Activity Recognitioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 4; 2026: 1.

12 papers.

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2024FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culturecited by 25
  3. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Modelscited by 14
  4. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  5. 2024Vision-Language Models under Cultural and Inclusive Considerationscited by 8
  6. 2025RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understandingcited by 8
  7. 2025Multimodal Cultural Safety: Evaluation Framework and Alignment Strategiescited by 8
  8. 2024Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Modelscited by 7
  9. 2024How Culturally Aware are Vision-Language Models?cited by 6
  10. 2025Evaluation of Cultural Competence of Vision-Language Modelscited by 3
  11. 2024Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Modelscited by 2
  12. 2026CulturalMenuBench: Probing the Knowledge-Application Gap in Multimodal Culinary Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
  2. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  3. 2025Seeing Culture: A Benchmark for Visual Reasoning and Grounding
  4. 2024Benchmarking Vision Language Models for Cultural Understanding
  5. 2025Rice-VL: Evaluating Vision-Language Models for Cultural Understanding Across ASEAN Countries
  6. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
  7. 2025Culture In a Frame: C3B as a Comic-Based Benchmark for Multimodal Culturally Awareness
  8. 2024CIC: A Framework for Culturally-Aware Image Captioning