mlsanityData to 7 Oct 2026

Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models

arXiv 2103.08849 ↗First posted 16 Mar 2021cs.CV cs.CLcited by 41

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This paper studies zero-shot cross-lingual transfer of vision-language models. Specifically, we focus on multilingual text-to-video search and propose a Transformer-based model that learns contextualized multilingual multimodal embeddings. Under a zero-shot setting, we empirically demonstrate that performance degrades significantly when we query the multilingual text-video model with non-English sentences. To address this problem, we introduce a multilingual multimodal pre-training strategy, and collect a new multilingual instructional video dataset (MultiHowTo100M) for pre-training. Experiments on VTT show that our method significantly improves video search in non-English languages without additional annotations. Furthermore, when multilingual annotations are available, our method outperforms recent baselines by a large margin in multilingual text-to-video search on VTT and VATEX; as well as in multilingual text-to-image search on Multi30K. Our model and Multi-HowTo100M is available at http://github.com/berniebear/Multi-HT100M.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 92 references that are in the map. First posted on arXiv 16 Mar 2021.

Builds on

57 of 92 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  10. 2018Deep contextualized word representationscited by 2,285
  11. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  12. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
38 more
  1. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  2. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  3. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  4. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
  5. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  6. 2019How multilingual is Multilingual BERT?cited by 642
  7. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  8. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  9. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  10. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  11. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  12. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  13. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  14. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  15. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  16. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  17. 2014Unifying Visual-Semantic Embeddings with Multimodal Neural Language Modelscited by 197
  18. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  19. 2020Support-set bottlenecks for video-text representation learningcited by 170
  20. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  21. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  22. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  23. 2020Multi-modal Dense Video Captioningcited by 113
  24. 2015Order-Embeddings of Images and Languagecited by 112
  25. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  26. 2019Fine-Grained Action Retrieval Through Multiple Parts-of-Speech Embeddingscited by 101
  27. 2019Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Taskscited by 96
  28. 2018Dual Encoding for Zero-Example Video Retrievalcited by 85
  29. 2018Rapid Adaptation of Neural Machine Translation to New Languagescited by 82
  30. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74
  31. 2019Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsingcited by 69
  32. 2020Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translationcited by 51
  33. 2016Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classificationcited by 48
  34. 2018A Corpus for Multilingual Document Classification in Eight Languagescited by 42
  35. 2015What's Cookin'? Interpreting Cooking Videos using Text, Speech and Visioncited by 31
  36. 2020Visual Grounding in Video for Unsupervised Word Translationcited by 27
  37. 2019MULE: Multimodal Universal Language Embeddingcited by 22
  38. 2020Learning to Scale Multilingual Representations for Vision-Language Taskscited by 22

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 15; 2023: 9; 2024: 8; 2025: 1; 2026: 2.

41 papers.

Sort:

By area:

  1. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  2. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  3. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  4. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  5. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  6. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  7. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  8. 2024SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languagescited by 41
  9. 2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
  10. 2022Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learningcited by 22
  11. 2022CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for Multimodal Sentiment Detectioncited by 21
  12. 2022Z-LaVI: Zero-Shot Language Solver Fueled by Visual Imaginationcited by 18
29 more
  1. 2024MultiVENT 2.0: A Massive Multilingual Benchmark for Event-Centric Video Retrievalcited by 17
  2. 2023CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translationcited by 14
  3. 2024Multimodal LLM Enhanced Cross-lingual Cross-modal Retrievalcited by 13
  4. 2023CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfercited by 10
  5. 2023Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal Retrievalcited by 9
  6. 2022Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-trainingcited by 7
  7. 2023MultiVENT: Multilingual Videos of Events with Aligned Natural Textcited by 7
  8. 2023TG-VQA: Ternary Game of Video Question Answeringcited by 6
  9. 2022Low-resource Neural Machine Translation with Cross-modal Alignmentcited by 5
  10. 2022VTC: Improving Video-Text Retrieval with User Commentscited by 5
  11. 2023FLAP: Fast Language-Audio Pre-trainingcited by 5
  12. 2024m3P: Towards Multimodal Multilingual Translation with Multimodal Promptcited by 5
  13. 2024Hierarchical Banzhaf Interaction for General Video-Language Representation Learningcited by 4
  14. 2022Generalizing Multimodal Pre-training into Multilingual via Language Acquisitioncited by 3
  15. 2022ChiQA: A Large Scale Image-based Real-World Question Answering Dataset for Multi-Modal Understandingcited by 3
  16. 2022MuMUR : Multilingual Multimodal Universal Retrievalcited by 3
  17. 2022C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrievalcited by 3
  18. 2022Vision-and-Language Pretrainingcited by 2
  19. 2024CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrievalcited by 2
  20. 2022cViL: Cross-Lingual Training of Vision-Language Models using Knowledge Distillationcited by 1
  21. 2022Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunitiescited by 1
  22. 2022Grafting Pre-trained Models for Multimodal Headline Generationcited by 1
  23. 2023Grounded and Well-rounded: A Methodological Approach to the Study of Cross-modal and Cross-lingual Groundingcited by 1
  24. 2023A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrievalcited by 1
  25. 2025Leveraging Auxiliary Information in Text-to-Video Retrieval: A Reviewcited by 1
  26. 2024Sparsely Multimodal Data Fusioncited by 0
  27. 2024Foundations of Multisensory Artificial Intelligencecited by 0
  28. 2026Why Vision Fails as a Universal Bridge: Rectifying Modality Asynchrony in Multilingual MLLMscited by 0
  29. 2026MultiVENT-Raw: A Benchmark for Retrieval and Reasoning over Raw Videoscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
  2. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  3. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages
  4. 2023Parameter-Efficient Cross-lingual Transfer of Vision and Language Models via Translation-based Alignment
  5. 2025Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model
  6. 2022Improving the Cross-Lingual Generalisation in Visual Question Answering
  7. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Data
  8. 2026Beyond Language Modeling: An Exploration of Multimodal Pretraining