mlsanityData to 7 Oct 2026

VL-BEiT: Generative Vision-Language Pretraining

arXiv 2206.01127 ↗First posted 2 Jun 2022cs.CV cs.CLcited by 39

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce a vision-language foundation model called VL-BEiT, which is a bidirectional multimodal Transformer learned by generative pretraining. Our minimalist solution conducts masked prediction on both monomodal and multimodal data with a shared Transformer. Specifically, we perform masked vision-language modeling on image-text pairs, masked language modeling on texts, and masked image modeling on images. VL-BEiT is learned from scratch with one unified pretraining task, one shared backbone, and one-stage training. Our method is conceptually simple and empirically effective. Experimental results show that VL-BEiT obtains strong results on various vision-language benchmarks, such as visual question answering, visual reasoning, and image-text retrieval. Moreover, our method learns transferable visual features, achieving competitive performance on image classification, and semantic segmentation.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 51 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

41 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  2. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts3 Nov
  3. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
26 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  10. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  11. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  15. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  16. 2016Deep Networks with Stochastic Depthcited by 973
  17. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  18. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  19. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  20. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  21. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  22. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  23. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  24. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  25. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  26. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 19; 2024: 10; 2025: 0; 2026: 2.

39 papers.

Sort:

By area:

  1. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  2. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  3. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  4. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  5. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  6. 2022A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyondcited by 58
  7. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  8. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 58
  9. 2023Improving Vision-and-Language Navigation by Generating Future-View Image Semanticscited by 46
  10. 2024Autoregressive Models in Vision: A Surveycited by 39
  11. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  12. 2023SLTUNET: A Simple Unified Model for Sign Language Translationcited by 35
27 more
  1. 2023CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservationcited by 29
  2. 2023Towards Unified Scene Text Spotting based on Sequence Generationcited by 27
  3. 2023EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoEcited by 24
  4. 2023Robust Visual Question Answering: Datasets, Methods, and Future Challengescited by 23
  5. 2024ProtChatGPT: Towards Understanding Proteins with Large Language Modelscited by 18
  6. 2022Seeing What You Miss: Vision-Language Pre-training with Semantic Completion Learningcited by 14
  7. 2023MAGVLT: Masked Generative Vision-and-Language Transformercited by 14
  8. 2023Global and Local Semantic Completion Learning for Vision-Language Pre-trainingcited by 12
  9. 2022Masked Contrastive Pre-Training for Efficient Video-Text Retrievalcited by 11
  10. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Taskscited by 9
  11. 2023Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Taskscited by 9
  12. 2023Accelerating Vision-Language Pretraining with Free Language Modelingcited by 9
  13. 2023BrainNPT: Pre-training of Transformer networks for brain network classificationcited by 9
  14. 2023Masked Autoencoding Does Not Help Natural Language Supervision at Scalecited by 7
  15. 2023Exploiting the Textual Potential from Vision-Language Pre-training for Text-based Person Searchcited by 6
  16. 2022Spatial Entropy as an Inductive Bias for Vision Transformerscited by 5
  17. 2023BARTPhoBEiT: Pre-trained Sequence-to-Sequence and Image Transformers Models for Vietnamese Visual Question Answeringcited by 5
  18. 2023Aerial Image Object Detection With Vision Transformer Detector (ViTDet)cited by 3
  19. 2024Calibrated Cache Model for Few-Shot Vision-Language Model Adaptationcited by 3
  20. 2026SkeletonContext: Skeleton-side Context Prompt Learning for Zero-Shot Skeleton-based Action Recognitioncited by 3
  21. 2026Let ViT Speak: Generative Language-Image Pre-trainingcited by 2
  22. 2023ArchBERT: Bi-Modal Understanding of Neural Architectures and Natural Languagescited by 1
  23. 2024Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-trainingcited by 1
  24. 2024A Survey of Low-shot Vision-Language Model Adaptation via Representer Theoremcited by 1
  25. 2024Compositional Kronecker Context Optimization for Vision-Language Modelscited by 0
  26. 2024ContextBLIP: Doubly Contextual Alignment for Contrastive Image Retrieval from Linguistically Complex Descriptionscited by 0
  27. 2024Braille-to-Speech Generator: Audio Generation Based on Joint Fine-Tuning of CLIP and Fastspeech2cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022VLMAE: Vision-Language Masked Autoencoder
  2. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
  3. 2022Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics
  4. 2022A Survey of Vision-Language Pre-Trained Models
  5. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning
  6. 2021Unified Multimodal Pre-training and Prompt-based Tuning for Vision-Language Understanding and Generation
  7. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
  8. 2023Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction