Vision-and-Language Pretraining
Not on the map. Why?
Abstract (arXiv)
With the burgeoning amount of data of image-text pairs and diversity of Vision-and-Language (V&L) tasks, scholars have introduced an abundance of deep learning models in this research domain. Furthermore, in recent years, transfer learning has also shown tremendous success in Computer Vision for tasks such as Image Classification, Object Detection, etc., and in Natural Language Processing for Question Answering, Machine Translation, etc. Inheriting the spirit of Transfer Learning, research works in V&L have devised multiple pretraining techniques on large-scale datasets in order to enhance the performance of downstream tasks. The aim of this article is to provide a comprehensive revision of contemporary V&L pretraining models. In particular, we categorize and delineate pretraining approaches, along with the summary of state-of-the-art vision-and-language pretrained models. Moreover, a list of training datasets and downstream tasks is supplied to further polish the perspective into V&L pretraining. Lastly, we decided to take a further step to discuss numerous directions for future research.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 99 of its 123 references that are in the map. First posted on arXiv 5 Jul 2022.
Builds on
99 of 123 references.
References · 99
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 2014Explaining and Harnessing Adversarial Examplescited by 7,287
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
38 more
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
- 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
- 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
- 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
- 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
- 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
- 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
- 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
- 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
- 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
- 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
- 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
- 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
- 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
- 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
- 2019Is Attention Interpretable?cited by 387
- 2019Using Pre-Training Can Improve Model Robustness and Uncertaintycited by 373
- 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
- 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
- 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
- 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
- 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
- 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
- 2018Bilinear Attention Networkscited by 277
- 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
- 2020UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learningcited by 239
- 2019EGNet:Edge Guidance Network for Salient Object Detectioncited by 237
- 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
- 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 213
- 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
- 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
- 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
50 most cited of 99 All 99 (JSON)
Built on by
Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1.
2 papers.