Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
Vision-Language Pre-training (VLP) has advanced the performance of many vision-language tasks, such as image-text retrieval, visual entailment, and visual reasoning. The pre-training mostly utilizes lexical databases and image queries in English. Previous work has demonstrated that the pre-training in English does not transfer well to other languages in a zero-shot setting. However, multilingual pre-trained language models (MPLM) have excelled at a variety of single-modal language tasks. In this paper, we propose a simple yet efficient approach to adapt VLP to unseen languages using MPLM. We utilize a cross-lingual contextualized token embeddings alignment approach to train text encoders for non-English languages. Our approach does not require image input and primarily uses machine translation, eliminating the need for target language data. Our evaluation across three distinct tasks (image-text retrieval, visual entailment, and natural language visual reasoning) demonstrates that this approach outperforms the state-of-the-art multilingual vision-language models without requiring large parallel corpora. Our code is available at https://github.com/Yasminekaroui/CliCoTea.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 34 references that are in the map. First posted on arXiv 29 Jun 2023.
Builds on
18 of 34 references.
References · 18
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
- 2019How multilingual is Multilingual BERT?cited by 645
- 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
- 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 223
- 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
6 more
- 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 171
- 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
- 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
- 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
- 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
- 2022Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-trainingcited by 24
Built on by
Citing papers per year: 2025: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2022Generalizing Multimodal Pre-training into Multilingual via Language Acquisition
- 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment
- 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
- 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
- 2024NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training
- 2022A Survey of Vision-Language Pre-Trained Models
- 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
- 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision