mlsanityData to 7 Oct 2026

Unified Vision-Language Pre-Training for Image Captioning and VQA

arXiv 1909.11059 ↗First posted 24 Sep 2019cs.CVcited by 494

Not on the map. Why?

Cited by 86 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper presents a unified Vision-Language Pre-training (VLP) model. The model is unified in that (1) it can be fine-tuned for either vision-language generation (e.g., image captioning) or understanding (e.g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models. The unified VLP model is pre-trained on a large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. The two tasks differ solely in what context the prediction conditions on. This is controlled by utilizing specific self-attention masks for the shared transformer network. To the best of our knowledge, VLP is the first reported model that achieves state-of-the-art results on both vision-language generation and understanding tasks, as disparate as image captioning and visual question answering, across three challenging benchmark datasets: COCO Captions, Flickr30k Captions, and VQA 2.0. The code and the pre-trained models are available at https://github.com/LuoweiZhou/VLP.

Key facts. Cited by 494 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 120; 2022: 138; 2023: 128; 2024: 62; 2025: 32; 2026: 14.

494 papers.

Sort:

By area:

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  2. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  7. 2021Grounded Language-Image Pre-trainingcited by 1,147
  8. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  9. 2021Transformers in Vision: A Surveycited by 1,009
  10. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  11. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  12. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
38 more
  1. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  2. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  3. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  4. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  5. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529
  6. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  7. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  8. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  9. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  10. 2022Multimodal Learning with Transformers: A Surveycited by 399
  11. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  12. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  13. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  14. 2022GLIPv2: Unifying Localization and Vision-Language Understandingcited by 275
  15. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  16. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  17. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  18. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  19. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  20. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  21. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  22. 2023Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?cited by 203
  23. 2023ULIP-2: Towards Scalable Multimodal Pre-training for 3D Understandingcited by 199
  24. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  25. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  26. 2023Recommender Systems in the Era of Large Language Models (LLMs)cited by 188
  27. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  28. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  29. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  30. 2021Chasing Sparsity in Vision Transformers: An End-to-End Explorationcited by 181
  31. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  32. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 175
  33. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  34. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  35. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  36. 2021VisualMRC: Machine Reading Comprehension on Document Imagescited by 150
  37. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  38. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146

50 most cited of 494 All 494 (JSON)