# From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models

arXiv: https://arxiv.org/abs/2407.00263 · HTML: https://mlsanity.com/paper/2407.00263/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 28 Jun 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Reproducible scaling laws for contrastive language-image learning](https://mlsanity.com/paper/2212.07143/) (2022)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [Towards Measuring the Representation of Subjective Global Opinions in Language Models](https://mlsanity.com/paper/2306.16388/) (2023)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Vision-Language Pre-Training with Triple Contrastive Learning](https://mlsanity.com/paper/2202.10401/) (2022)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World](https://mlsanity.com/paper/1711.08536/) (2017)
- [Does Object Recognition Work for Everyone?](https://mlsanity.com/paper/1906.02659/) (2019)
- [Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision](https://mlsanity.com/paper/2202.08360/) (2022)
- [Extracting Cultural Commonsense Knowledge at Scale](https://mlsanity.com/paper/2210.07763/) (2022)
- [Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning](https://mlsanity.com/paper/2109.06860/) (2021)
- [GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition](https://mlsanity.com/paper/2301.02560/) (2023)
- [Multi-lingual and Multi-cultural Figurative Language Understanding](https://mlsanity.com/paper/2305.16171/) (2023)
- [Inspecting the Geographical Representativeness of Images from Text-to-Image Models](https://mlsanity.com/paper/2305.11080/) (2023)
- [GRIT: General Robust Image Task Benchmark](https://mlsanity.com/paper/2204.13653/) (2022)
- [GeoNet: Benchmarking Unsupervised Adaptation across Geographies](https://mlsanity.com/paper/2303.15443/) (2023)
- [GD-COMET: A Geo-Diverse Commonsense Inference Model](https://mlsanity.com/paper/2310.15383/) (2023)
- [Towards A Fairer Landmark Recognition Dataset](https://mlsanity.com/paper/2108.08874/) (2021)

## Built on by (most cited first, 14 of 14)

- [RAVENEA: A Benchmark for Multimodal Retrieval-Augmented Visual Culture Understanding](https://mlsanity.com/paper/2505.14462/) (2025)
- [Vision-Language Models under Cultural and Inclusive Considerations](https://mlsanity.com/paper/2407.06177/) (2024)
- [Pearl: A Multimodal Culturally-Aware Arabic Instruction Dataset](https://mlsanity.com/paper/2505.21979/) (2025)
- [What are Foundation Models Cooking in the Post-Soviet World?](https://mlsanity.com/paper/2502.18583/) (2025)
- [CultureCLIP: Empowering CLIP with Cultural Awareness through Synthetic Images and Contextualized Captions](https://mlsanity.com/paper/2507.06210/) (2025)
- [Cultural Awareness in Vision-Language Models: A Cross-Country Exploration](https://mlsanity.com/paper/2505.20326/) (2025)
- [Culture In a Frame: C3B as a Comic-Based Benchmark for Multimodal Culturally Awareness](https://mlsanity.com/paper/2510.00041/) (2025)
- [Rice-VL: Evaluating Vision-Language Models for Cultural Understanding Across ASEAN Countries](https://mlsanity.com/paper/2512.01419/) (2025)
- [Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation](https://mlsanity.com/paper/2508.16762/) (2025)
- [Are Vision Language Models Cross-Cultural Theory of Mind Reasoners?](https://mlsanity.com/paper/2512.17394/) (2025)
- [From Perception to Punchline: Empowering VLM with the Art of In-the-wild Meme](https://mlsanity.com/paper/2512.24555/) (2025)
- [CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generation](https://mlsanity.com/paper/2505.04851/) (2025)
- [EgoGroups: A Benchmark For Detecting Social Groups of People in the Wild](https://mlsanity.com/paper/2603.22249/) (2026)
- [Beyond Semantics: Modeling Factual and Affective Perceptual Experiences from Vision-Language Data](https://mlsanity.com/paper/2606.03345/) (2026)

## Abstract (arXiv)

Despite recent advancements in vision-language models, their performance remains suboptimal on images from non-western cultures due to underrepresentation in training datasets. Various benchmarks have been proposed to test models' cultural inclusivity, but they have limited coverage of cultures and do not adequately assess cultural diversity across universal as well as culture-specific local concepts. To address these limitations, we introduce the GlobalRG benchmark, comprising two challenging tasks: retrieval across universals and cultural visual grounding. The former task entails retrieving culturally diverse images for universal concepts from 50 countries, while the latter aims at grounding culture-specific concepts within images from 15 countries. Our evaluation across a wide range of models reveals that the performance varies significantly across cultures -- underscoring the necessity for enhancing multicultural understanding in vision-language models.