mlsanityData to 7 Oct 2026

GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods

arXiv 2301.01893 ↗First posted 5 Jan 2023cs.CV cs.AI cs.CLcited by 11

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

A key goal for the advancement of AI is to develop technologies that serve the needs not just of one group but of all communities regardless of their geographical region. In fact, a significant proportion of knowledge is locally shared by people from certain regions but may not apply equally in other regions because of cultural differences. If a model is unaware of regional characteristics, it may lead to performance disparity across regions and result in bias against underrepresented groups. We propose GIVL, a Geographically Inclusive Vision-and-Language Pre-trained model. There are two attributes of geo-diverse visual concepts which can help to learn geo-diverse knowledge: 1) concepts under similar categories have unique knowledge and visual characteristics, 2) concepts with similar visual features may fall in completely different categories. Motivated by the attributes, we design new pre-training objectives Image Knowledge Matching (IKM) and Image Edit Checking (IEC) to pre-train GIVL. Compared with similar-size models pre-trained with similar scale of data, GIVL achieves state-of-the-art (SOTA) and more balanced performance on geo-diverse V&L tasks.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 45 of its 63 references that are in the map. First posted on arXiv 5 Jan 2023.

Builds on

45 of 63 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment29 Aug
  2. 2022VLMixer: Unpaired Vision-Language Pre-training via Cross-Modal CutMix17 Jun
  3. 2022VLUE: A Multi-Task Benchmark for Evaluating Vision-Language Models30 May
  4. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  5. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug
  6. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  10. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
27 more
  1. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  5. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  6. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  7. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  8. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  9. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  10. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  11. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  12. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  13. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  14. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  15. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  16. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  17. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  18. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  19. 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
  20. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  21. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 152
  22. 2019Does Object Recognition Work for Everyone?cited by 149
  23. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  24. 2021Broaden the Vision: Geo-Diverse Visual Commonsense Reasoningcited by 59
  25. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  26. 2022GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Modelscited by 30
  27. 2022Can domain adaptation make object recognition work for everyone?cited by 8

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 6; 2025: 1.

11 papers.

  1. 2023Inspecting the Geographical Representativeness of Images from Text-to-Image Modelscited by 44
  2. 2024Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaborationcited by 19
  3. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  4. 2023GD-COMET: A Geo-Diverse Commonsense Inference Modelcited by 13
  5. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Conceptscited by 11
  6. 2025Culture in Action: Evaluating Text-to-Image Models through Social Activitiescited by 9
  7. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  8. 2024CIC: A Framework for Culturally-Aware Image Captioningcited by 8
  9. 2023A Survey on Image-text Multimodal Modelscited by 4
  10. 2024Incorporating Geo-Diverse Knowledge into Prompting for Increased Geographical Robustness in Object Recognitioncited by 4
  11. 2024Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Modelscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models
  2. 2026Anthropogenic Regional Adaptation in Multimodal Vision-Language Model
  3. 2025GS-Bias: Global-Spatial Bias Learner for Single-Image Test-Time Adaptation of Vision-Language Models
  4. 2025SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data
  5. 2024Toward Interactive Regional Understanding in Vision-Large Language Models
  6. 2024RegionGPT: Towards Region Understanding Vision Language Model
  7. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
  8. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models