mlsanityData to 7 Oct 2026

Masked Feature Prediction for Self-Supervised Visual Pre-Training

arXiv 2112.09133 ↗First posted 16 Dec 2021cs.CV cs.LGcited by 573

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Masked Feature Prediction (MaskFeat) for self-supervised pre-training of video models. Our approach first randomly masks out a portion of the input sequence and then predicts the feature of the masked regions. We study five different types of features and find Histograms of Oriented Gradients (HOG), a hand-crafted feature descriptor, works particularly well in terms of both performance and efficiency. We observe that the local contrast normalization in HOG is essential for good results, which is in line with earlier work using HOG for visual recognition. Our approach can learn abundant visual knowledge and drive large-scale Transformer-based models. Without using extra model weights or supervision, MaskFeat pre-trained on unlabeled videos achieves unprecedented results of 86.7% with MViT-L on Kinetics-400, 88.3% on Kinetics-600, 80.4% on Kinetics-700, 39.8 mAP on AVA, and 75.0% on SSv2. MaskFeat further generalizes to image input, which can be interpreted as a video with a single frame and obtains competitive results on ImageNet.

Key facts. Cited by 573 papers in the map (data through 7 Oct 2026). Builds on 84 of its 99 references that are in the map. First posted on arXiv 16 Dec 2021.

Builds on

84 of 99 references.

References · 84

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  9. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  6. 2017Neural Discrete Representation Learningcited by 4,650
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  10. 2016Improved Techniques for Training GANscited by 2,938
  11. 2015Fast R-CNNcited by 2,930
  12. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  13. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  14. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  15. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  16. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  17. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  18. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  19. 2016Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Networkcited by 1,898
  20. 2017The Kinetics Human Action Video Datasetcited by 1,894
  21. 2018SlowFast Networks for Video Recognitioncited by 1,862
  22. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  23. 2021ViViT: A Video Vision Transformercited by 1,610
  24. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  25. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  26. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  27. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  28. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  29. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  30. 2019Contrastive Multiview Codingcited by 1,297
  31. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  32. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
  33. 2016Colorful Image Colorizationcited by 1,113
  34. 2018Self-Attention with Relative Position Representationscited by 1,112
  35. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  36. 2021Video Swin Transformercited by 1,090
  37. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  38. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020

50 most cited of 84 All 84 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 157; 2023: 183; 2024: 112; 2025: 76; 2026: 45.

573 papers.

Sort:

By area:

  1. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2022Token Merging: Your ViT But Fastercited by 995
  4. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
  5. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  6. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  7. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  8. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  9. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  10. 2022Masked Autoencoders for Point Cloud Self-supervised Learningcited by 456
  11. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  12. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
38 more
  1. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  2. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 315
  3. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  4. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  5. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  6. 2023RoMa: Robust Dense Feature Matchingcited by 296
  7. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  8. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  9. 2022Masked Autoencoders that Listencited by 271
  10. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  11. 2022MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesiscited by 222
  12. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  13. 2023Video Understanding with Large Language Models: A Surveycited by 203
  14. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  15. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  16. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  17. 2023EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anythingcited by 187
  18. 2022Frozen CLIP Models are Efficient Video Learnerscited by 181
  19. 2022MIC: Masked Image Consistency for Context-Enhanced Domain Adaptationcited by 176
  20. 2022Masked World Models for Visual Controlcited by 169
  21. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  22. 2022CroCo v2: Improved Cross-view Completion Pre-training for Stereo Matching and Optical Flowcited by 165
  23. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 162
  24. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  25. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  26. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155
  27. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  28. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  29. 2022Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decodingcited by 142
  30. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  31. 2023GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learnercited by 131
  32. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  33. 2022Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning?cited by 121
  34. 20234M: Massively Multimodal Masked Modelingcited by 118
  35. 2022ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyondcited by 117
  36. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  37. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  38. 2022SemMAE: Semantic-Guided Masking for Learning Masked Autoencoderscited by 113

50 most cited of 573 All 573 (JSON)