# Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders

arXiv: https://arxiv.org/abs/2504.21681 · HTML: https://mlsanity.com/paper/2504.21681/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 30 Apr 2025.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [Language-agnostic BERT Sentence Embedding](https://mlsanity.com/paper/2007.01852/) (2020)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://mlsanity.com/paper/2004.09813/) (2020)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation](https://mlsanity.com/paper/2004.11867/) (2020)
- [Visually Grounded Reasoning across Languages and Cultures](https://mlsanity.com/paper/2109.13238/) (2021)
- [Word Alignment by Fine-tuning Embeddings on Parallel Corpora](https://mlsanity.com/paper/2101.08231/) (2021)
- [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://mlsanity.com/paper/2205.06266/) (2022)
- [Multilingual Alignment of Contextual Word Representations](https://mlsanity.com/paper/2002.03518/) (2020)
- [Masked Vision and Language Modeling for Multi-modal Representation Learning](https://mlsanity.com/paper/2208.02131/) (2022)
- [UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training](https://mlsanity.com/paper/2104.00332/) (2021)
- [Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation](https://mlsanity.com/paper/2205.12647/) (2022)
- [BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning](https://mlsanity.com/paper/2206.08657/) (2022)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs](https://mlsanity.com/paper/2307.06930/) (2023)
- [M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks](https://mlsanity.com/paper/2407.03791/) (2024)
- [Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training](https://mlsanity.com/paper/2206.00621/) (2022)
- [Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages](https://mlsanity.com/paper/2306.16774/) (2023)

## Built on by (most cited first, 1 of 1)

- [Multilingual Vision-Language Models, A Survey](https://mlsanity.com/paper/2509.22123/) (2025)

## Abstract (arXiv)

Most pre-trained Vision-Language (VL) models and training data for the downstream tasks are only available in English. Therefore, multilingual VL tasks are solved using cross-lingual transfer: fine-tune a multilingual pre-trained model or transfer the text encoder using parallel data. We study the alternative approach: transferring an already trained encoder using parallel data. We investigate the effect of parallel data: domain and the number of languages, which were out of focus in previous work. Our results show that even machine-translated task data are the best on average, caption-like authentic parallel data outperformed it in some languages. Further, we show that most languages benefit from multilingual training.