Data to 7 Oct 2026

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

arXiv 1908.06066 ↗First posted 16 Aug 2019cs.CVcited by 451

Not on the map. Why?

Cited by 85 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose Unicoder-VL, a universal encoder that aims to learn joint representations of vision and language in a pre-training manner. Borrow ideas from cross-lingual pre-trained models, such as XLM and Unicoder, both visual and linguistic contents are fed into a multi-layer Transformer for the cross-modal pre-training, where three pre-trained tasks are employed, including Masked Language Modeling (MLM), Masked Object Classification (MOC) and Visual-linguistic Matching (VLM). The first two tasks learn context-aware representations for input tokens based on linguistic and visual contents jointly. The last task tries to predict whether an image and a text describe each other. After pretraining on large-scale image-caption pairs, we transfer Unicoder-VL to caption-based image-text retrieval and visual commonsense reasoning, with just one additional output layer. We achieve state-of-the-art or comparable results on both two tasks and show the powerful ability of the cross-modal pre-training.

Key facts. Cited by 451 papers in the map (data through 7 Oct 2026). First posted on arXiv 16 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 123; 2022: 152; 2023: 108; 2024: 48; 2025: 18; 2026: 2.

451 papers.

Sort:

By area:

2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
2021Grounded Language-Image Pre-trainingcited by 1,147
2021Transformers in Vision: A Surveycited by 1,009
2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
2022Language-driven Semantic Segmentationcited by 711
2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484

38 more

2022Multimodal Learning with Transformers: A Surveycited by 399
2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 336
2021Pre-Trained Models: Past, Present and Futurecited by 319
2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 224
2021A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Modelcited by 223
2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
2021General Facial Representation Learning in a Visual-Linguistic Mannercited by 193
2021Visually Grounded Reasoning across Languages and Culturescited by 188
2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
2022A Survey of Vision-Language Pre-Trained Modelscited by 177
2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
2023Towards Open Vocabulary Learning: A Surveycited by 165
2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
2023ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Promptscited by 160
2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
2021Towards Long-Form Video Understandingcited by 146
2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124

50 most cited of 451 All 451 (JSON)