# Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models

arXiv: https://arxiv.org/abs/2407.02067 · HTML: https://mlsanity.com/paper/2407.02067/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [Open-Vocabulary Object Detection Using Captions](https://mlsanity.com/paper/2011.10678/) (2020)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [TRAK: Attributing Model Behavior at Scale](https://mlsanity.com/paper/2303.14186/) (2023)
- [Challenges and Strategies in Cross-Cultural NLP](https://mlsanity.com/paper/2203.10020/) (2022)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Stability](https://mlsanity.com/paper/1310.0150/) (2013)
- [Tag2Text: Guiding Vision-Language Model via Image Tagging](https://mlsanity.com/paper/2303.05657/) (2023)
- [Zero-shot Image Editing with Reference Imitation](https://mlsanity.com/paper/2406.07547/) (2024)
- [Massively Multi-Cultural Knowledge Acquisition & LM Benchmarking](https://mlsanity.com/paper/2402.09369/) (2024)
- [An image speaks a thousand words, but can everyone listen? On image transcreation for cultural relevance](https://mlsanity.com/paper/2404.01247/) (2024)
- [ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generation](https://mlsanity.com/paper/2401.06310/) (2024)
- [Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art](https://mlsanity.com/paper/2406.03930/) (2024)
- [How Culture Shapes What People Want From AI](https://mlsanity.com/paper/2403.05104/) (2024)
- [BertaQA: How Much Do Language Models Know About Local Culture?](https://mlsanity.com/paper/2406.07302/) (2024)
- [VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models](https://mlsanity.com/paper/2404.13874/) (2024)
- [No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models](https://mlsanity.com/paper/2405.13777/) (2024)
- [Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing](https://mlsanity.com/paper/2402.06015/) (2024)
- [GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods](https://mlsanity.com/paper/2301.01893/) (2023)
- [Generalized People Diversity: Learning a Human Perception-Aligned Diversity Representation for People Images](https://mlsanity.com/paper/2401.14322/) (2024)

## Built on by (most cited first, 2 of 2)

- [Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models](https://mlsanity.com/paper/2502.14906/) (2025)
- [Vision-Language Models under Cultural and Inclusive Considerations](https://mlsanity.com/paper/2407.06177/) (2024)

## Abstract (arXiv)

We present a comprehensive three-phase study to examine (1) the cultural understanding of Large Multimodal Models (LMMs) by introducing DalleStreet, a large-scale dataset generated by DALL-E 3 and validated by humans, containing 9,935 images of 67 countries and 10 concept classes; (2) the underlying implicit and potentially stereotypical cultural associations with a cultural artifact extraction task; and (3) an approach to adapt cultural representation in an image based on extracted associations using a modular pipeline, CultureAdapt. We find disparities in cultural understanding at geographic sub-region levels with both open-source (LLaVA) and closed-source (GPT-4V) models on DalleStreet and other existing benchmarks, which we try to understand using over 18,000 artifacts that we identify in association to different countries. Our findings reveal a nuanced picture of the cultural competence of LMMs, highlighting the need to develop culture-aware systems. Dataset and code are available at https://github.com/iamshnoo/crossroads