A Survey of Vision-Language Pre-training from the Lens of Multimodal Machine Translation
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Large language models such as BERT and the GPT series started a paradigm shift that calls for building general-purpose models via pre-training on large datasets, followed by fine-tuning on task-specific datasets. There is now a plethora of large pre-trained models for Natural Language Processing and Computer Vision. Recently, we have seen rapid developments in the joint Vision-Language space as well, where pre-trained models such as CLIP (Radford et al., 2021) have demonstrated improvements in downstream tasks like image captioning and visual question answering. However, surprisingly there is comparatively little work on exploring these models for the task of multimodal machine translation, where the goal is to leverage image/video modality in text-to-text translation. To fill this gap, this paper surveys the landscape of language-and-vision pre-training from the lens of multimodal machine translation. We summarize the common architectures, pre-training objectives, and datasets from literature and conjecture what further is needed to make progress on multimodal machine translation.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 50 references that are in the map. First posted on arXiv 12 Jun 2023.
Builds on
37 of 50 references.
Closest ancestors · 4
Same direction, newest first
- 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
- 2022A Survey of Vision-Language Pre-Trained Models18 Feb
- 2022VLP: A Survey on Vision-Language Pre-training18 Feb
- 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
Other references · 33
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
21 more
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
- 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
- 2019Cross-lingual Language Model Pretrainingcited by 1,090
- 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
- 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
- 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
- 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
- 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
- 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 493
- 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
- 2020VirTex: Learning Visual Representations from Textual Annotationscited by 339
- 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
- 2015YFCC100M: The New Data in Multimedia Researchcited by 235
- 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
- 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 138
- 2020Learning Visual Representations with Caption Annotationscited by 123
- 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
- 2019Multimodal Machine Translation through Visuals and Speechcited by 29
- 2021Cross-lingual Visual Pre-training for Multimodal Machine Translationcited by 19
- 2022VISA: An Ambiguous Subtitles Dataset for Visual Scene-Aware Machine Translationcited by 11
Built on by
Citing papers per year: 2024: 1; 2025: 1.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
- 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
- 2021Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
- 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
- 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
- 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Survey
- 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
- 2025Dual-branch Prompting for Multimodal Machine Translation