mlsanityData to 7 Oct 2026

See It from My Perspective: How Language Affects Cultural Bias in Image Understanding

arXiv 2406.11665 ↗First posted 17 Jun 2024cs.CL cs.AI cs.CVcited by 17

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) can respond to queries about images in many languages. However, beyond language, culture affects how we see things. For example, individuals from Western cultures focus more on the central figure in an image while individuals from East Asian cultures attend more to scene context. In this work, we characterize the Western bias of VLMs in image understanding and investigate the role that language plays in this disparity. We evaluate VLMs across subjective and objective visual tasks with culturally diverse images and annotations. We find that VLMs perform better on the Western split than on the East Asian split of each task. Through controlled experimentation, we trace one source of this bias in image understanding to the lack of diversity in language model construction. While inference in a language nearer to a culture can lead to reductions in bias, we show it is much more effective when that language was well-represented during text-only pre-training. Interestingly, this yields bias reductions even when prompting in English. Our work highlights the importance of richer representation of all languages in building equitable VLMs.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 44 of its 67 references that are in the map. First posted on arXiv 17 Jun 2024.

Builds on

44 of 67 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun
  2. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models22 May
  3. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases3 Apr
  4. 2024Open-ended VQA benchmarking of Vision-Language models by exploiting Classification datasets and their semantic hierarchy11 Feb
  5. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs13 Jul
  6. 2023GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods5 Jan

Other references · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Segment Anythingcited by 8,158
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  10. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
26 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  3. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  4. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  5. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  6. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 275
  7. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  8. 2023Having Beer after Prayer? Measuring Cultural Bias in Large Language Modelscited by 153
  9. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 152
  10. 2019Does Object Recognition Work for Everyone?cited by 149
  11. 2024Investigating Cultural Alignment of Large Language Modelscited by 140
  12. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  13. 2021Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Modelscited by 132
  14. 2022Language Models as Agent Modelscited by 131
  15. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  16. 2021ArtEmis: Affective Language for Visual Artcited by 103
  17. 2023NLPositionality: Characterizing Design Biases of Datasets and Modelscited by 91
  18. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  19. 2021Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metricscited by 83
  20. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 82
  21. 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
  22. 2022ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culturecited by 17
  23. 2022Multilingual Multimodal Learning with Machine Translated Textcited by 16
  24. 2023Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?cited by 13
  25. 2023Bridging the Digital Divide: Performance Variation across Socio-Economic Factors in Vision-Language Modelscited by 11
  26. 2024What Is Missing in Multilingual Visual Reasoning and How to Fix Itcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 7.

17 papers.

Sort:

By area:

  1. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  2. 2025OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QAcited by 5
  3. 2025Uncovering Cultural Representation Disparities in Vision-Language Modelscited by 4
  4. 2025Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understandingcited by 3
  5. 2025Seeing Symbols, Missing Cultures: Probing Vision-Language Models' Reasoning on Fire Imagery and Cultural Meaningcited by 3
  6. 2025Finding Culture-Sensitive Neurons in Vision-Language Modelscited by 3
  7. 2025Cultural Awareness in Vision-Language Models: A Cross-Country Explorationcited by 2
  8. 2026From Native Memes to Global Moderation: Cross-Cultural Evaluation of Vision-Language Models for Hateful Meme Detectioncited by 2
  9. 2025The Describe-Then-Generate Bottleneck: How VLM Descriptions Alter Image Generation Outcomescited by 1
  10. 2025On The Conceptualization and Societal Impact of Cross-Cultural Biascited by 1
  11. 2026CArtBench: Evaluating Vision-Language Models on Chinese Art Understanding, Interpretation, and Authenticitycited by 1
  12. 2026Vision-Language Models Suppress Female Representations Under Ambiguous Inputcited by 1
5 more
  1. 2025Mining Contextualized Visual Associations from Images for Creativity Understandingcited by 0
  2. 2026MiSCHiEF: A Benchmark in Minimal-Pairs of Safety and Culture for Holistic Evaluation of Fine-Grained Image-Caption Alignmentcited by 0
  3. 2026AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagerycited by 0
  4. 2026Uncertainty-Aware Art-Historical Dating with Vision-Language Modelscited by 0
  5. 2026MemeCULT-1K: Benchmarking South Asian Cultural Context and Humor Understanding of Multimodal Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models
  2. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
  3. 2025Contrasting Cognitive Styles in Vision-Language Models: Holistic Attention in Japanese Versus Analytical Focus in English
  4. 2025Multilingual Vision-Language Models, A Survey
  5. 2024Can We Talk Models Into Seeing the World Differently?
  6. 2025Vision Language Models are Confused Tourists
  7. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  8. 2025Evaluation of Cultural Competence of Vision-Language Models