Data to 7 Oct 2026

BEiT: BERT Pre-Training of Image Transformers

arXiv 2106.08254 ↗First posted 15 Jun 2021cs.CV cs.LGcited by 2,360

Not on the map. Why?

Cited by 113 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a self-supervised vision representation model BEiT, which stands for Bidirectional Encoder representation from Image Transformers. Following BERT developed in the natural language processing area, we propose a masked image modeling task to pretrain vision Transformers. Specifically, each image has two views in our pre-training, i.e, image patches (such as 16x16 pixels), and visual tokens (i.e., discrete tokens). We first "tokenize" the original image into visual tokens. Then we randomly mask some image patches and fed them into the backbone Transformer. The pre-training objective is to recover the original visual tokens based on the corrupted image patches. After pre-training BEiT, we directly fine-tune the model parameters on downstream tasks by appending task layers upon the pretrained encoder. Experimental results on image classification and semantic segmentation show that our model achieves competitive results with previous pre-training methods. For example, base-size BEiT achieves 83.2% top-1 accuracy on ImageNet-1K, significantly outperforming from-scratch DeiT training (81.8%) with the same setup. Moreover, large-size BEiT obtains 86.3% only using ImageNet-1K, even outperforming ViT-L with supervised pre-training on ImageNet-22K (85.2%). The code and pretrained models are available at https://aka.ms/beit.

Key facts. Cited by 2,360 papers in the map (data through 7 Oct 2026). Builds on 55 of its 59 references that are in the map. First posted on arXiv 15 Jun 2021.

Builds on

55 of 59 references.

References · 55

2017Attention Is All You Needcited by 59,932
2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
2014Adam: A Method for Stochastic Optimizationcited by 36,960
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
2014Microsoft COCO: Common Objects in Contextcited by 17,498
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
2018Representation Learning with Contrastive Predictive Codingcited by 7,662
2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482

38 more

2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
2017Neural Discrete Representation Learningcited by 4,650
2020Training data-efficient image transformers & distillation through attentioncited by 4,560
2021Zero-Shot Text-to-Image Generationcited by 4,276
2013Auto-Encoding Variational Bayescited by 3,980
2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
2020Exploring Simple Siamese Representation Learningcited by 2,853
2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
2016Colorful Image Colorizationcited by 1,113
2019Cross-lingual Language Model Pretrainingcited by 1,088
2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
2021Scaling Vision Transformerscited by 993
2016Deep Networks with Stochastic Depthcited by 973
2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
2021Going deeper with Image Transformerscited by 759
2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
2020Prototypical Contrastive Learning of Unsupervised Representationscited by 636
2019Self-labelling via simultaneous clustering and representation learningcited by 525
2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 62; 2022: 503; 2023: 663; 2024: 521; 2025: 389; 2026: 222.

2,360 papers.

Sort:

By area:

2023Segment Anythingcited by 8,158
2021Evaluating Large Language Models Trained on Codecited by 8,053
2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
2022A ConvNet for the 2020scited by 3,545
2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 1,981
2022Visual Prompt Tuningcited by 1,674
2024Depth Anything V2cited by 1,385
2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346

38 more

2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
2022MaskGIT: Masked Generative Image Transformercited by 1,082
2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
2023Vision Transformers Need Registerscited by 844
2023Vision-Language Models for Vision Tasks: A Surveycited by 834
2022Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Imagescited by 823
2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
2021Vector-quantized Image Modeling with Improved VQGANcited by 656
2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
2023ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depthcited by 646
2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
2023One Fits All:Power General Time Series Analysis by Pretrained LMcited by 615
2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
2022Vision Transformer Adapter for Dense Predictionscited by 564
2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 481
2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
2024MOMENT: A Family of Open Time-series Foundation Modelscited by 453
2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 450
2023Generative Multimodal Models are In-Context Learnerscited by 446
2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443

50 most cited of 2,360 All 2,360 (JSON)