mlsanityData to 14 Oct 2026

Vision Language Models are Confused Tourists

arXiv 2511.17004 ↗First posted 21 Nov 2025cs.CV cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Although the cultural dimension has been one of the key aspects in evaluating Vision-Language Models (VLMs), their ability to remain stable across diverse cultural inputs remains largely untested, despite being crucial to support diversity and multicultural societies. Existing evaluations often rely on benchmarks featuring only a singular cultural concept per image, overlooking scenarios where multiple, potentially unrelated cultural cues coexist. To address this gap, we introduce ConfusedTourist, a novel cultural adversarial robustness suite designed to assess VLMs' stability against perturbed geographical cues. Our experiments reveal a critical vulnerability, where accuracy drops heavily under simple image-stacking perturbations and even worsens with its image-generation-based variant. Interpretability analyses further show that these failures stem from systematic attention shifts toward distracting cues, diverting the model from its intended focus. These findings highlight a critical challenge: visual cultural concept mixing can substantially impair even state-of-the-art VLMs, underscoring the urgent need for more culturally robust multimodal understanding.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 26 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

19 of 26 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Seeing Culture: A Benchmark for Visual Reasoning and Grounding20 Sep
  2. 2025Vision Language Models are Biased29 May
  3. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries2 Jan
  4. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages25 Nov
  5. 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
  6. 2024Vision language models are blind: Failing to translate detailed visual features into words9 Jul
  7. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun

Other references · 12

  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  5. 2021Understanding Robustness of Transformers for Image Classificationcited by 249
  6. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
  7. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 43
  8. 2024Evaluating Morphological Compositional Generalization in Large Language Modelscited by 19
  9. 2025CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Interventioncited by 11
  10. 2024CultiVerse: Towards Cross-Cultural Understanding for Paintings with Large Language Modelcited by 5
  11. 2025Evaluating Compositional Generalisation in VLMs and Diffusion Modelscited by 5
  12. 2025Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretabilitycited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Sci-Rho: A Multilingual Visually-Grounded Symbolic Benchmark for STEM Problemscited by 0
  2. 2026Cultural Moment Benchmark: Evaluating Video Cultural Reasoning and Grounding in Southeast Asiacited by 0
  3. 2026In-Context Learning Can Help Vision-Language Models Overcome Training Priorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
  2. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding
  3. 2025Evaluation of Cultural Competence of Vision-Language Models
  4. 2025Uncovering Cultural Representation Disparities in Vision-Language Models
  5. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
  6. 2025Toward Socially Aware Vision-Language Models: Evaluating Cultural Competence Through Multimodal Story Generation
  7. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
  8. 2025Exposing Blindspots: Cultural Bias Evaluation in Generative Image Models