mlsanityData to 14 Oct 2026

No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models

arXiv 2405.13777 ↗First posted 22 May 2024cs.CV cs.AIcited by 16

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We study cultural and socioeconomic diversity in contrastive vision-language models (VLMs). Using a broad range of benchmark datasets and evaluation metrics, we bring to attention several important findings. First, the common filtering of training data to English image-text pairs disadvantages communities of lower socioeconomic status and negatively impacts cultural understanding. Notably, this performance gap is not captured by - and even at odds with - the currently popular evaluation metrics derived from the Western-centric ImageNet and COCO datasets. Second, pretraining with global, unfiltered data before fine-tuning on English content can improve cultural understanding without sacrificing performance on said popular benchmarks. Third, we introduce the task of geo-localization as a novel evaluation metric to assess cultural diversity in VLMs. Our work underscores the value of using diverse data to create more inclusive multimodal systems and lays the groundwork for developing VLMs that better represent global perspectives.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 78 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

65 of 78 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023PaLI-3 Vision Language Models: Smaller, Faster, Stronger13 Oct
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  10. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  12. 2019Natural Adversarial Examplescited by 1,408
38 more
  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  2. 2018Model Cards for Model Reportingcited by 1,080
  3. 2021Scaling Vision Transformerscited by 999
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  5. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  6. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  7. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  9. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  10. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  11. 2020Are we done with ImageNet?cited by 333
  12. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  13. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 309
  14. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  15. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  16. 2023Demystifying CLIP Datacited by 270
  17. 2023Data Filtering Networkscited by 250
  18. 2022Patching open-vocabulary models by interpolating weightscited by 190
  19. 2021Visually Grounded Reasoning across Languages and Culturescited by 189
  20. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  21. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 153
  22. 2019Does Object Recognition Work for Everyone?cited by 151
  23. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  24. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  25. 2021Systematic Inequalities in Language Technology Performance across the World's Languagescited by 120
  26. 2022Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Datasetcited by 100
  27. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  28. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  29. 2023Do Multilingual Language Models Think Better in English?cited by 88
  30. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  31. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  32. 2021Adaptive Methods for Real-World Domain Generalizationcited by 66
  33. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  34. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  35. 2023Uncurated Image-Text Datasets: Shedding Light on Demographic Biascited by 52
  36. 2023GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognitioncited by 50
  37. 2024CLIP the Bias: How Useful is Balancing Data in Multimodal Learning?cited by 35
  38. 2024LocCa: Visual Pretraining with Location-aware Captionerscited by 35

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 10; 2026: 1.

16 papers.

Sort:

By area:

  1. 2025Meta CLIP 2: A Worldwide Scaling Recipecited by 64
  2. 2025FineVision: Open Data Is All You Needcited by 53
  3. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  4. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 23
  5. 2025MobileCLIP2: Improving Multi-Modal Reinforced Trainingcited by 21
  6. 2025Scaling Pre-training to One Hundred Billion Data for Vision Language Modelscited by 19
  7. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  8. 2025MixMin: Finding Data Mixtures via Convex Minimizationcited by 11
  9. 2025Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systemscited by 10
  10. 2024BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrievalcited by 6
  11. 2025What Are You Doing? A Closer Look at Controllable Human Video Generationcited by 3
  12. 2025OpenCity3D: What do Vision-Language Models know about Urban Environments?cited by 3
4 more
  1. 2024Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Modelscited by 2
  2. 2025Cultural Awareness in Vision-Language Models: A Cross-Country Explorationcited by 2
  3. 2025Data Matters Most: Auditing Social Bias in Contrastive Vision Language Modelscited by 1
  4. 2026Unlocking Cognitive Capabilities and Analyzing the Perception-Logic Trade-offcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models
  2. 2025Uncovering Cultural Representation Disparities in Vision-Language Models
  3. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  4. 2025Multilingual Vision-Language Models, A Survey
  5. 2024Benchmarking Vision Language Models for Cultural Understanding
  6. 2025Rice-VL: Evaluating Vision-Language Models for Cultural Understanding Across ASEAN Countries
  7. 2025Vision Language Models are Confused Tourists
  8. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models