mlsanityData to 7 Oct 2026

Vision-Language Models under Cultural and Inclusive Considerations

arXiv 2407.06177 ↗First posted 8 Jul 2024cs.CV cs.AI cs.CLcited by 8

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large vision-language models (VLMs) can assist visually impaired people by describing images from their daily lives. Current evaluation datasets may not reflect diverse cultural user backgrounds or the situational context of this use case. To address this problem, we create a survey to determine caption preferences and propose a culture-centric evaluation benchmark by filtering VizWiz, an existing dataset with images taken by people who are blind. We then evaluate several VLMs, investigating their reliability as visual assistants in a culturally diverse setting. While our results for state-of-the-art models are promising, we identify challenges such as hallucination and misalignment of automatic evaluation metrics with human judgment. We make our survey, data, code, and model outputs publicly available.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 51 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

19 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models2 Jul
  2. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models28 Jun
  3. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun
  4. 2024How Culturally Aware are Vision-Language Models?24 May
  5. 2024What matters when building vision-language models?3 May
  6. 2024Exploring Visual Culture Awareness in GPT-4V: A Comprehensive Probing8 Feb

Other references · 13

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Mistral 7Bcited by 3,065
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  6. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  7. 2020Captioning Images Taken by People Who Are Blindcited by 108
  8. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  9. 2024Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision Peoplecited by 24
  10. 2023Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Modelscited by 16
  11. 2023Towards Equitable Representation in Text-to-Image Synthesis Models with the Cross-Cultural Understanding Benchmark (CCUB) Datasetcited by 12
  12. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
1 more
  1. 2023Towards Real Time Egocentric Segment Captioning for The Blind and Visually Impaired in RGB-D Theatre Imagescited by 2

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 6.

8 papers.

  1. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  2. 2025Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Userscited by 11
  3. 2025Provocations from the Humanities for Generative AI Researchcited by 8
  4. 2024Multi3Hate: Multimodal, Multilingual, and Multicultural Hate Speech Detection with Vision-Language Modelscited by 7
  5. 2025What are Foundation Models Cooking in the Post-Soviet World?cited by 3
  6. 2025Evaluation of Cultural Competence of Vision-Language Modelscited by 3
  7. 2025Culture In a Frame: C3B as a Comic-Based Benchmark for Multimodal Culturally Awarenesscited by 2
  8. 2025Multilingual Vision-Language Models, A Surveycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Are Large Vision-Language Models Ready to Guide Blind and Low-Vision Individuals?
  2. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  3. 2025How Blind and Low-Vision Individuals Prefer Large Vision-Language Model-Generated Scene Descriptions
  4. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models
  5. 2024Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration
  6. 2025Uncovering Cultural Representation Disparities in Vision-Language Models
  7. 2024Benchmarking Vision Language Models for Cultural Understanding
  8. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding