mlsanityData to 7 Oct 2026

Multilingual Multimodal Learning with Machine Translated Text

arXiv 2210.13134 ↗First posted 24 Oct 2022cs.CL cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

Most vision-and-language pretraining research focuses on English tasks. However, the creation of multilingual multimodal evaluation datasets (e.g. Multi30K, xGQA, XVNLI, and MaRVL) poses a new challenge in finding high-quality training data that is both multilingual and multimodal. In this paper, we investigate whether machine translating English multimodal data can be an effective proxy for the lack of readily available multilingual data. We call this framework TD-MML: Translated Data for Multilingual Multimodal Learning, and it can be applied to any multimodal dataset and model. We apply it to both pretraining and fine-tuning data with a state-of-the-art model. In order to prevent models from learning from low-quality translated text, we propose two metrics for automatically removing such translations from the resulting datasets. In experiments on five tasks across 20 languages in the IGLUE benchmark, we show that translated data can provide a useful signal for multilingual multimodal learning, both at pretraining and fine-tuning.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 24 Oct 2022.

Builds on

28 of 40 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
16 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  3. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  4. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  5. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  6. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  7. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  8. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  9. 2021Visually Grounded Reasoning across Languages and Culturescited by 188
  10. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  11. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  12. 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
  13. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  14. 2021Cross-lingual Visual Pre-training for Multimodal Machine Translationcited by 19
  15. 2017Image Pivoting for Learning Multilingual Multimodal Representationscited by 17
  16. 2018Lessons learned in multilingual grounded language learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 5; 2025: 4; 2026: 2.

16 papers.

Sort:

By area:

  1. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languagescited by 52
  2. 2023LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Promptingcited by 17
  3. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understandingcited by 17
  4. 2024No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Modelscited by 16
  5. 2024Why do LLaVA Vision-Language Models Reply to Images in English?cited by 11
  6. 2023Semantic and Expressive Variation in Image Captions Across Languagescited by 8
  7. 2025Breaking Language Barriers in Visual Language Models via Multilingual Textual Regularizationcited by 4
  8. 2023Meta-learning For Vision-and-language Cross-lingual Transfercited by 3
  9. 2025POLYCHARTQA: Benchmarking Large Vision-Language Models with Multilingual Chart Question Answeringcited by 3
  10. 2024Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answeringcited by 2
  11. 2023ICU: Conquering Language Barriers in Vision-and-Language Modeling by Dividing the Tasks into Image Captioning and Language Understandingcited by 1
  12. 2024Improving the Consistency in Cross-Lingual Cross-Modal Retrieval with 1-to-K Contrastive Learningcited by 1
4 more
  1. 2025uCLIP: Parameter-Efficient Multilingual Extension of Vision-Language Models with Unpaired Datacited by 1
  2. 2026Large Language Models for Mental Health: A Multilingual Evaluationcited by 1
  3. 2025Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translationcited by 0
  4. 2026On the Limitations of Cross-Lingual Consistency in Multilingual Text-to-image Generationcited by 0