mlsanityData to 7 Oct 2026

MVP: Multimodality-guided Visual Pre-training

arXiv 2203.05175 ↗First posted 10 Mar 2022cs.CVcited by 92

Not on the map. Why?

Abstract (arXiv)

Recently, masked image modeling (MIM) has become a promising direction for visual pre-training. In the context of vision transformers, MIM learns effective visual representation by aligning the token-level features with a pre-defined space (e.g., BEIT used a d-VAE trained on a large image corpus as the tokenizer). In this paper, we go one step further by introducing guidance from other modalities and validating that such additional knowledge leads to impressive gains for visual pre-training. The proposed approach is named Multimodality-guided Visual Pre-training (MVP), in which we replace the tokenizer with the vision branch of CLIP, a vision-language model pre-trained on 400 million image-text pairs. We demonstrate the effectiveness of MVP by performing standard experiments, i.e., pre-training the ViT models on ImageNet and fine-tuning them on a series of downstream visual recognition tasks. In particular, pre-training ViT-Base/16 for 300 epochs, MVP reports a 52.4% mIoU on ADE20K, surpassing BEIT (the baseline and previous state-of-the-art) with an impressive margin of 6.8%.

Key facts. Cited by 92 papers in the map (data through 7 Oct 2026). Builds on 34 of its 39 references that are in the map. First posted on arXiv 10 Mar 2022.

Builds on

34 of 39 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2017Neural Discrete Representation Learningcited by 4,650
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
22 more
  1. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  2. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  3. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  4. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  5. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  6. 2019Contrastive Multiview Codingcited by 1,297
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  11. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  12. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  13. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  14. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  15. 2018Rethinking ImageNet Pre-trainingcited by 376
  16. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  17. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  18. 2021PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformerscited by 204
  19. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  20. 2019D-VAE: A Variational Autoencoder for Directed Acyclic Graphscited by 98
  21. 2018Iterative Reorganization with Weak Spatial Constraints: Solving Arbitrary Jigsaw Puzzles for Unsupervised Representation Learningcited by 51
  22. 2021Exploring the Diversity and Invariance in Yourself for Visual Pre-Training Taskcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 18; 2023: 43; 2024: 17; 2025: 9; 2026: 5.

92 papers.

Sort:

By area:

  1. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  2. 2023EVA-02: A Visual Representation for Neon Genesiscited by 377
  3. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  4. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  5. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  6. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  7. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  8. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  9. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  10. 2023CLIP-KD: An Empirical Study of CLIP Model Distillationcited by 87
  11. 2022An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modelingcited by 74
  12. 2023Diffusion Models as Masked Autoencoderscited by 71
38 more
  1. 2022MILAN: Masked Image Pretraining on Language Assisted Representationcited by 69
  2. 2022Fine-Grained Semantically Aligned Vision-Language Pre-Trainingcited by 68
  3. 2022MAViL: Masked Audio-Video Learnerscited by 56
  4. 2022Exploring Target Representations for Masked Autoencoderscited by 46
  5. 2022Masked Image Modeling with Denoising Contrastcited by 45
  6. 2023What Makes for Good Visual Tokenizers for Large Language Models?cited by 42
  7. 2023Continual Vision-Language Representation Learning with Off-Diagonal Informationcited by 40
  8. 2022A Unified View of Masked Image Modelingcited by 36
  9. 2023Decoupling Common and Unique Representations for Multimodal Self-supervised Learningcited by 36
  10. 2022CLIP Itself is a Strong Fine-tuner: Achieving 85.7% and 88.0% Top-1 Accuracy with ViT-B and ViT-L on ImageNetcited by 35
  11. 2023HAP: Structure-Aware Masked Image Modeling for Human-Centric Perceptioncited by 33
  12. 2023Contrastive Feature Masking Open-Vocabulary Vision Transformercited by 31
  13. 2022Stare at What You See: Masked Image Modeling without Reconstructioncited by 30
  14. 2023Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptationcited by 30
  15. 2023DIME-FM: DIstilling Multimodal and Efficient Foundation Modelscited by 28
  16. 2023Degeneration-Tuning: Using Scrambled Grid shield Unwanted Concepts from Stable Diffusioncited by 27
  17. 2022CAE v2: Context Autoencoder with CLIP Targetcited by 25
  18. 2022Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migrationcited by 24
  19. 2023SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformerscited by 24
  20. 2024UNIC: Universal Classification Models via Multi-teacher Distillationcited by 24
  21. 2023Empowering Low-Light Image Enhancer through Customized Learnable Priorscited by 22
  22. 2024EVA-X: A Foundation Model for General Chest X-ray Analysis with Self-supervised Learningcited by 22
  23. 2023Gradient-Regulated Meta-Prompt Learning for Generalizable Vision-Language Modelscited by 21
  24. 2023Towards Enhanced Image Inpainting: Mitigating Unwanted Object Insertion and Preserving Color Consistencycited by 19
  25. 2024Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learningcited by 19
  26. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  27. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understandingcited by 17
  28. 2024ViTGaze: Gaze Following with Interaction Features in Vision Transformerscited by 15
  29. 2023VPP: Efficient Conditional 3D Generation via Voxel-Point Progressive Representationcited by 14
  30. 2023Masked Modeling for Self-supervised Representation Learning on Vision and Beyondcited by 13
  31. 2023Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimickingcited by 12
  32. 2024Denoising with a Joint-Embedding Predictive Architecturecited by 12
  33. 2023RILS: Masked Visual Reconstruction in Language Semantic Spacecited by 10
  34. 2023Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Taskscited by 9
  35. 2023MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignmentscited by 9
  36. 2024ComKD-CLIP: Comprehensive Knowledge Distillation for Contrastive Language-Image Pre-traning Modelcited by 9
  37. 2022Towards Sustainable Self-supervised Learningcited by 8
  38. 2024QPT V2: Masked Image Modeling Advances Visual Scoringcited by 8

50 most cited of 92 All 92 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022VL-BEiT: Generative Vision-Language Pretraining
  2. 2022A Survey of Vision-Language Pre-Trained Models
  3. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
  4. 2022Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends
  6. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
  7. 2023ViLTA: Enhancing Vision-Language Pre-training through Textual Augmentation
  8. 2025OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning