# Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing

arXiv: https://arxiv.org/abs/2402.06015 · HTML: https://mlsanity.com/paper/2402.06015/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 17 of 17)

- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Is ChatGPT a Good NLG Evaluator? A Preliminary Study](https://mlsanity.com/paper/2303.04048/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Improving CLIP Training with Language Rewrites](https://mlsanity.com/paper/2305.20088/) (2023)
- [Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study](https://mlsanity.com/paper/2303.17466/) (2023)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://mlsanity.com/paper/2203.13722/) (2022)
- [The Ghost in the Machine has an American accent: value conflict in GPT-3](https://mlsanity.com/paper/2203.07785/) (2022)
- [mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs](https://mlsanity.com/paper/2307.06930/) (2023)
- [VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation](https://mlsanity.com/paper/2312.09251/) (2023)
- [Contrastive Alignment of Vision to Language Through Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2303.11866/) (2023)
- [Semantic and Expressive Variation in Image Captions Across Languages](https://mlsanity.com/paper/2310.14356/) (2023)
- [Modelling the Influence of Cultural Information on Vision-Based Human Home Activity Recognition](https://mlsanity.com/paper/1803.07915/) (2018)

## Built on by (most cited first, 12 of 12)

- [Survey of Cultural Awareness in Language Models: Text and Beyond](https://mlsanity.com/paper/2411.00860/) (2024)
- [FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culture](https://mlsanity.com/paper/2406.11030/) (2024)
- [Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models](https://mlsanity.com/paper/2502.14906/) (2025)
- [CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts](https://mlsanity.com/paper/2410.15453/) (2024)
- [RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding](https://mlsanity.com/paper/2505.14462/) (2025)
- [Vision-Language Models under Cultural and Inclusive Considerations](https://mlsanity.com/paper/2407.06177/) (2024)
- [Multimodal Cultural Safety: Evaluation Framework and Alignment Strategies](https://mlsanity.com/paper/2505.14972/) (2025)
- [Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Models](https://mlsanity.com/paper/2411.03888/) (2024)
- [How Culturally Aware are Vision-Language Models?](https://mlsanity.com/paper/2405.17475/) (2024)
- [Evaluation of Cultural Competence of Vision-Language Models](https://mlsanity.com/paper/2505.22793/) (2025)
- [Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models](https://mlsanity.com/paper/2407.02067/) (2024)
- [CulturalMenuBench: Probing the Knowledge-Application Gap in Multimodal Culinary Reasoning](https://mlsanity.com/paper/2609.03526/) (2026)

## Abstract (arXiv)

Pretrained large Vision-Language models have drawn considerable interest in recent years due to their remarkable performance. Despite considerable efforts to assess these models from diverse perspectives, the extent of visual cultural awareness in the state-of-the-art GPT-4V model remains unexplored. To tackle this gap, we extensively probed GPT-4V using the MaRVL benchmark dataset, aiming to investigate its capabilities and limitations in visual understanding with a focus on cultural aspects. Specifically, we introduced three visual related tasks, i.e. caption classification, pairwise captioning, and culture tag selection, to systematically delve into fine-grained visual cultural evaluation. Experimental results indicate that GPT-4V excels at identifying cultural concepts but still exhibits weaker performance in low-resource languages, such as Tamil and Swahili. Notably, through human evaluation, GPT-4V proves to be more culturally relevant in image captioning tasks than the original MaRVL human annotations, suggesting a promising solution for future visual cultural benchmark construction.