mlsanityData to 7 Oct 2026

Cross-lingual Visual Pre-training for Multimodal Machine Translation

arXiv 2101.10044 ↗First posted 25 Jan 2021cs.CL cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

Pre-trained language models have been shown to improve performance in many natural language tasks substantially. Although the early focus of such models was single language pre-training, recent advances have resulted in cross-lingual and visual pre-training methods. In this paper, we combine these two approaches to learn visually-grounded cross-lingual representations. Specifically, we extend the translation language modelling (Lample and Conneau, 2019) with masked region classification and perform pre-training with three-way parallel vision & language corpora. We show that when fine-tuned for multimodal machine translation, these models obtain state-of-the-art performance. We also provide qualitative insights into the usefulness of the learned grounded representations.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 25 Jan 2021.

Builds on

33 of 45 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  7. 2018Deep contextualized word representationscited by 2,285
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2019Language Models as Knowledge Bases?cited by 1,922
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
21 more
  1. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  2. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  3. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  6. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  7. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  8. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  9. 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
  10. 2019On Identifiability in Transformerscited by 115
  11. 201975 Languages, 1 Model: Parsing Universal Dependencies Universallycited by 90
  12. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
  13. 2020A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translationcited by 51
  14. 2017Imagination improves Multimodal Translationcited by 38
  15. 2017Attention Strategies for Multi-Source Sequence-to-Sequence Learningcited by 32
  16. 2019Multimodal Machine Translation through Visuals and Speechcited by 29
  17. 2018The MeMAD Submission to the WMT18 Multimodal Translation Taskcited by 22
  18. 2018CUNI System for the WMT18 Multimodal Translation Taskcited by 13
  19. 2016Does Multimodality Help Human and Machine for Translation and Image Captioning?cited by 7
  20. 2017LIUM-CVC Submissions for WMT17 Multimodal Translation Taskcited by 7
  21. 2018UMONS Submission for WMT18 Multimodal Translation Taskcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 11; 2023: 4; 2024: 3; 2025: 1.

19 papers.

Sort:

By area:

  1. 2022IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languagescited by 54
  2. 2022On Vision Features in Multimodal Machine Translationcited by 31
  3. 2022VALHALLA: Visual Hallucination for Machine Translationcited by 27
  4. 2022Neural Machine Translation with Phrase-Level Universal Visual Representationscited by 26
  5. 2022Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluationcited by 26
  6. 2022Multilingual Multimodal Learning with Machine Translated Textcited by 16
  7. 2022Increasing Visual Awareness in Multimodal Neural Machine Translation from an Information Theoretic Perspectivecited by 10
  8. 2023BigVideo: A Large-scale Video Subtitle Translation Dataset for Multimodal Machine Translationcited by 9
  9. 2022Beyond Triplet: Leveraging the Most Data for Multimodal Machine Translationcited by 7
  10. 2024A Survey on Multi-modal Machine Translation: Tasks, Methods and Challengescited by 6
  11. 2022Low-resource Neural Machine Translation with Cross-modal Alignmentcited by 5
  12. 2022DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attentioncited by 4
7 more
  1. 2024Adding Multimodal Capabilities to a Text-only Translation Modelcited by 4
  2. 2023Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering Pairscited by 3
  3. 2024Towards Zero-Shot Multimodal Machine Translationcited by 3
  4. 2023A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translationcited by 2
  5. 2022Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunitiescited by 1
  6. 2023Generalization algorithm of multimodal pre-training model based on graph-text self-supervised trainingcited by 0
  7. 2025Memory Reviving, Continuing Learning and Beyond: Evaluation of Pre-trained Encoders and Decoders for Multimodal Machine Translationcited by 0