mlsanityData to 7 Oct 2026

Benchmarking Vision Language Models for Cultural Understanding

arXiv 2407.10920 ↗First posted 15 Jul 2024cs.CV cs.AI cs.CLcited by 100

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Foundation models and vision-language pre-training have notably advanced Vision Language Models (VLMs), enabling multimodal processing of visual and linguistic data. However, their performance has been typically assessed on general scene understanding - recognizing objects, attributes, and actions - rather than cultural comprehension. This study introduces CulturalVQA, a visual question-answering benchmark aimed at assessing VLM's geo-diverse cultural understanding. We curate a collection of 2,378 image-question pairs with 1-5 answers per question representing cultures from 11 countries across 5 continents. The questions probe understanding of various facets of culture such as clothing, food, drinks, rituals, and traditions. Benchmarking VLMs on CulturalVQA, including GPT-4V and Gemini, reveals disparity in their level of cultural understanding across regions, with strong cultural understanding capabilities for North America while significantly lower performance for Africa. We observe disparity in their performance across cultural facets too, with clothing, rituals, and traditions seeing higher performances than food and drink. These disparities help us identify areas where VLMs lack cultural understanding and demonstrate the potential of CulturalVQA as a comprehensive evaluation set for gauging VLM progress in understanding diverse cultures.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 39 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

20 of 39 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024PaliGemma: A versatile 3B VLM for transfer10 Jul
  2. 2024Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration24 Jun
  3. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun
  4. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs13 Jul
  5. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  7. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  8. 2022Challenges and Strategies in Cross-Cultural NLPcited by 216
  9. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  10. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  11. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  12. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
3 more
  1. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  2. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  3. 2024FoodieQA: A Multimodal Dataset for Fine-Grained Understanding of Chinese Food Culturecited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 46; 2026: 41.

100 papers.

Sort:

By area:

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  3. 2024GEOBench-VLM: Benchmarking Vision-Language Models for Geospatial Taskscited by 50
  4. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 46
  5. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countriescited by 40
  6. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisinescited by 39
  7. 2024CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingcited by 27
  8. 2024LLM-GLOBE: A Benchmark Evaluating the Cultural Values Embedded in LLM Outputcited by 27
  9. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  10. 2025CULEMO: Cultural Lenses on Emotion -- Benchmarking LLMs for Cross-Cultural Emotion Understandingcited by 14
  11. 2025MAKIEval: A Multilingual Automatic WiKidata-based Framework for Cultural Awareness Evaluation for LLMscited by 13
  12. 2025DRISHTIKON: A Multimodal Multilingual Benchmark for Testing Language Models' Understanding on Indian Culturecited by 13
38 more
  1. 2025Hire Your Anthropologist! Rethinking Culture Benchmarks Through an Anthropological Lenscited by 13
  2. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  3. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  4. 2025MMBoundary: Advancing MLLM Knowledge Boundary Awareness through Reasoning Step Confidence Calibrationcited by 11
  5. 2025CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMscited by 11
  6. 2025Time Travel: A Comprehensive Benchmark to Evaluate LMMs on Historical and Cultural Artifactscited by 10
  7. 2025Culture in Action: Evaluating Text-to-Image Models through Social Activitiescited by 9
  8. 2024Cultural Adaptation of Menus: A Fine-Grained Approachcited by 8
  9. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  10. 2025SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabiacited by 8
  11. 2025RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understandingcited by 8
  12. 2024Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Modelscited by 7
  13. 2024DARE: Diverse Visual Question Answering with Robustness Evaluationcited by 6
  14. 2026Tears or Cheers? Benchmarking LLMs via Culturally Elicited Distinct Affective Responsescited by 6
  15. 2026Cognitive Mismatch in Multimodal Large Language Models for Discrete Symbol Understandingcited by 6
  16. 2025Hanfu-Bench: A Multimodal Benchmark on Cross-Temporal Cultural Understanding and Transcreationcited by 5
  17. 2025OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QAcited by 5
  18. 2024KAHANI: Culturally-Nuanced Visual Storytelling Tool for Non-Western Culturescited by 4
  19. 2024Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domainscited by 4
  20. 2024SAFE-MEME: Structured Reasoning Framework for Robust Hate Speech Detection in Memescited by 4
  21. 2025TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMscited by 4
  22. 2025Uncovering Cultural Representation Disparities in Vision-Language Modelscited by 4
  23. 2025CaMMT: Benchmarking Culturally Aware Multimodal Machine Translationcited by 4
  24. 2025CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Datasetcited by 4
  25. 2025Isolating Culture Neurons in Multilingual Large Language Modelscited by 4
  26. 2025A Structured Framework for Evaluating and Enhancing Interpretive Capabilities of Multimodal LLMs in Culturally Situated Taskscited by 4
  27. 2025Exposing Blindspots: Cultural Bias Evaluation in Generative Image Modelscited by 4
  28. 2025When Tom Eats Kimchi: Evaluating Cultural Bias of Multimodal Large Language Models in Cultural Mixture Contextscited by 3
  29. 2025Traveling Across Languages: Benchmarking Cross-Lingual Consistency in Multimodal LLMscited by 3
  30. 2025Evaluation of Cultural Competence of Vision-Language Modelscited by 3
  31. 2025CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captionscited by 3
  32. 2025Seeing Symbols, Missing Cultures: Probing Vision-Language Models' Reasoning on Fire Imagery and Cultural Meaningcited by 3
  33. 2025Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sportscited by 3
  34. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Modelscited by 3
  35. 2025Finding Culture-Sensitive Neurons in Vision-Language Modelscited by 3
  36. 2025World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Modelscited by 3
  37. 2026Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialectscited by 3
  38. 2025Benchmarking Vision Language Models on German Factual Datacited by 2

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Seeing Culture: A Benchmark for Visual Reasoning and Grounding
  2. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
  3. 2025IndicVisionBench: Benchmarking Cultural and Multilingual Understanding in VLMs
  4. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledge
  5. 2024CVLUE: A New Benchmark Dataset for Chinese Vision-Language Understanding Evaluation
  6. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
  7. 2026ChinaHeritaQA: A Culturally-Grounded Visual Question Answering Dataset for World Heritage Sites in China
  8. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding