mlsanityData to 13 Oct 2026

BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers

arXiv 2208.06366 ↗First posted 12 Aug 2022cs.CVcited by 298

Not on the map. Why?

Cited by 31 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Masked image modeling (MIM) has demonstrated impressive results in self-supervised representation learning by recovering corrupted image patches. However, most existing studies operate on low-level image pixels, which hinders the exploitation of high-level semantics for representation models. In this work, we propose to use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing a systematic way to promote MIM from pixel-level to semantic-level. Specifically, we propose vector-quantized knowledge distillation to train the tokenizer, which discretizes a continuous semantic space to compact codes. We then pretrain vision Transformers by predicting the original visual tokens for the masked image patches. Furthermore, we introduce a patch aggregation strategy which associates discrete image patches to enhance global semantic representation. Experiments on image classification and semantic segmentation show that BEiT v2 outperforms all compared MIM methods. On ImageNet-1K (224 size), the base-size BEiT v2 achieves 85.5% top-1 accuracy for fine-tuning and 80.1% top-1 accuracy for linear probing. The large-size BEiT v2 obtains 87.3% top-1 accuracy for ImageNet-1K (224 size) fine-tuning, and 56.7% mIoU on ADE20K for semantic segmentation. The code and pretrained models are available at https://aka.ms/beitv2.

Key facts. Cited by 298 papers in the map (data through 13 Oct 2026). Builds on 29 of its 32 references that are in the map. First posted on arXiv 12 Aug 2022.

Builds on

29 of 32 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,043
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,364
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  7. 2017Neural Discrete Representation Learningcited by 4,680
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,237
  10. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,368
  11. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,847
  12. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,612
17 more
  1. 2019Natural Adversarial Examplescited by 1,405
  2. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,268
  3. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,219
  4. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,068
  5. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,012
  6. 2021Scaling Vision Transformerscited by 998
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 721
  8. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  9. 2021Vector-quantized Image Modeling with Improved VQGANcited by 657
  10. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 574
  11. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  12. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  13. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  14. 2021Condenser: a Pre-training Architecture for Dense Retrievalcited by 162
  15. 2021Are Large-scale Datasets Necessary for Self-Supervised Pre-training?cited by 117
  16. 2022MVP: Multimodality-guided Visual Pre-trainingcited by 92
  17. 2022Corrupted Image Modeling for Self-Supervised Visual Pre-Trainingcited by 68

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 20; 2023: 101; 2024: 82; 2025: 66; 2026: 29.

298 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,168
  2. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 689
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 627
  4. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  5. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 451
  6. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 406
  7. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 398
  8. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 384
  9. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  10. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Modelscited by 372
  11. 2024Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCIcited by 321
  12. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
38 more
  1. 2023A Cookbook of Self-Supervised Learningcited by 249
  2. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  3. 2024Rotary Position Embedding for Vision Transformercited by 230
  4. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 225
  5. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  6. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 202
  7. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  8. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  9. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  10. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  11. 2023MiDaS v3.1 -- A Model Zoo for Robust Monocular Relative Depth Estimationcited by 173
  12. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 167
  13. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  14. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  15. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  16. 20234M: Massively Multimodal Masked Modelingcited by 120
  17. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  18. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  19. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  20. 2024Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detectioncited by 103
  21. 2024Dinomaly: The Less Is More Philosophy in Multi-Class Unsupervised Anomaly Detectioncited by 93
  22. 2024ImageFolder: Autoregressive Image Generation with Folded Tokenscited by 93
  23. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  24. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  25. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87
  26. 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 75
  27. 2023Diffusion Models as Masked Autoencoderscited by 71
  28. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  29. 2024ViT-CoMer: Vision Transformer with Convolutional Multi-scale Feature Interaction for Dense Predictionscited by 66
  30. 2023A Billion-scale Foundation Model for Remote Sensing Imagescited by 62
  31. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  32. 2024EEGFormer: Towards Transferable and Interpretable Large-Scale EEG Foundation Modelcited by 51
  33. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encodingcited by 44
  34. 2022Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNetcited by 41
  35. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  36. 2024Autoregressive Models in Vision: A Surveycited by 39
  37. 2024SOLO: A Single Transformer for Scalable Vision-Language Modelingcited by 37
  38. 2022A Unified View of Masked Image Modelingcited by 36

50 most cited of 298 All 298 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026StableVQ: Practical Guidelines for Stable Vector-Quantized Tokenizer Training
  2. 2024Scalable Image Tokenization with Index Backpropagation Quantization
  3. 2023Masked Vector Quantization
  4. 2025Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Image Generation
  5. 2025Hybrid Token Compression for Vision-Language Models
  6. 2026Vision Foundation Models as Generalist Tokenizers for Image Generation
  7. 2026MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging
  8. 2024Factorized Visual Tokenization and Generation