mlsanityData to 14 Oct 2026

Generalizing Multimodal Pre-training into Multilingual via Language Acquisition

arXiv 2206.11091 ↗First posted 29 May 2022cs.CL cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

English-based Vision-Language Pre-training (VLP) has achieved great success in various downstream tasks. Some efforts have been taken to generalize this success to non-English languages through Multilingual Vision-Language Pre-training (M-VLP). However, due to the large number of languages, M-VLP models often require huge computing resources and cannot be flexibly extended to new languages. In this work, we propose a MultiLingual Acquisition (MLA) framework that can easily generalize a monolingual Vision-Language Pre-training model into multilingual. Specifically, we design a lightweight language acquisition encoder based on state-of-the-art monolingual VLP models. We further propose a two-stage training strategy to optimize the language acquisition encoder, namely the Native Language Transfer stage and the Language Exposure stage. With much less multilingual training data and computing resources, our model achieves state-of-the-art performance on multilingual image-text and video-text retrieval benchmarks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 48 references that are in the map. First posted on arXiv 29 May 2022.

Builds on

33 of 48 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models16 Mar

Other references · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  11. 2019Natural Adversarial Examplescited by 1,408
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
20 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  2. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  4. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  5. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  6. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  7. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  8. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  9. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  10. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  11. 2021WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Trainingcited by 84
  12. 2018COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrievalcited by 80
  13. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  14. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  15. 2017STAIR Captions: Constructing a Large-Scale Japanese Image Caption Datasetcited by 47
  16. 2020Towards Zero-shot Cross-lingual Image Retrievalcited by 25
  17. 2019MULE: Multimodal Universal Language Embeddingcited by 22
  18. 2020Learning to Scale Multilingual Representations for Vision-Language Taskscited by 22
  19. 2017Image Pivoting for Learning Multilingual Multimodal Representationscited by 17
  20. 2021Product-oriented Machine Translation with Cross-modal Cross-lingual Pre-trainingcited by 10

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2.

3 papers.

  1. 2022AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilitiescited by 85
  2. 2023Babel-ImageNet: Massively Multilingual Evaluation of Vision-and-Language Representationscited by 11
  3. 2023CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfercited by 10

Similar, not cited

Close in wording, no citation either way

  1. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  2. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  3. 2025Generalizing vision-language models to novel domains: A comprehensive survey
  4. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction
  5. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
  6. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  7. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
  8. 2022VLP: A Survey on Vision-Language Pre-training