mlsanityData to 14 Oct 2026

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision

arXiv 2108.10904 ↗First posted 24 Aug 2021cs.CV cs.CL cs.LGcited by 647

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the scalability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-specific objectives. In this work, we relax these constraints and present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM). Unlike prior work, SimVLM reduces the training complexity by exploiting large-scale weak supervision, and is trained end-to-end with a single prefix language modeling objective. Without utilizing extra data or task-specific customization, the resulting model significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA (+3.74% vqa-score), NLVR2 (+1.17% accuracy), SNLI-VE (+1.37% accuracy) and image captioning tasks (+10.1% average CIDEr score). Furthermore, we demonstrate that SimVLM acquires strong generalization and transfer ability, enabling zero-shot behavior including open-ended visual question answering and cross-modality transfer.

Key facts. Cited by 647 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 56 of its 66 references that are in the map. First posted on arXiv 24 Aug 2021.

Builds on

56 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning3 Jun
  2. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning7 Apr

Other references · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  8. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,288
38 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  3. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  12. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  13. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  14. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  15. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  16. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  17. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  18. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  19. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  20. 2021Transformers in Vision: A Surveycited by 1,013
  21. 2016Deep Networks with Stochastic Depthcited by 979
  22. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  23. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  24. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  25. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 569
  26. 2016Self-critical Sequence Training for Image Captioningcited by 547
  27. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  28. 2018nocaps: novel object captioning at scalecited by 444
  29. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  30. 2021Early Convolutions Help Transformers See Bettercited by 396
  31. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  32. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  33. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  34. 2019Attention on Attention for Image Captioningcited by 289
  35. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 287
  36. 2016Using the Output Embedding to Improve Language Modelscited by 272
  37. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 252
  38. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 29; 2022: 164; 2023: 235; 2024: 125; 2025: 76; 2026: 18.

647 papers.

Sort:

By area:

  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  7. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  10. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  12. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
38 more
  1. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
  2. 2023A Survey on Multimodal Large Language Modelscited by 909
  3. 2022A Generalist Agentcited by 808
  4. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  5. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  6. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  8. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  9. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  10. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  11. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  13. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  14. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  15. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  16. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  17. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  18. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  19. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  20. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  21. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  22. 2022Multimodal Learning with Transformers: A Surveycited by 400
  23. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  24. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  25. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  26. 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 308
  27. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  28. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  29. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 290
  30. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  31. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  32. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  33. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  34. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  35. 2023A Cookbook of Self-Supervised Learningcited by 250
  36. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  37. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  38. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220

50 most cited of 647 All 647 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Generalizing vision-language models to novel domains: A comprehensive survey
  2. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  3. 2023Vision-Language Models for Vision Tasks: A Survey
  4. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  5. 2024An Introduction to Vision-Language Modeling
  6. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Models
  7. 2024Unveiling Encoder-Free Vision-Language Models
  8. 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey